[00:00.000 --> 00:01.180]  I'm sitting there.
[00:01.380 --> 00:01.940]  I see.
[00:12.020 --> 00:13.460]  Yeah, it helps if you're on stage.
[00:19.070 --> 00:20.330]  None of these guys have slept.
[00:21.070 --> 00:22.530]  How many hours of sleep have you had?
[00:22.610 --> 00:22.910]  Oh, plenty.
[00:23.190 --> 00:24.130]  It's not the first day.
[00:24.570 --> 00:24.910]  Gotcha.
[00:25.470 --> 00:29.070]  Alright, well, I guess I'll... you set this up since you've done it for me.
[00:32.770 --> 00:34.130]  Dragorn, back in the house.
[00:34.350 --> 00:36.010]  Yeah, I'm always here, aren't I?
[00:37.150 --> 00:37.510]  Cool.
[00:38.530 --> 00:39.530]  Can I just give us a couple seconds?
[00:39.530 --> 00:40.490]  Oh, that's cool.
[00:40.690 --> 00:44.090]  Now, so, I'm just going to let you guys introduce yourselves.
[00:44.290 --> 00:44.370]  Right.
[00:46.650 --> 00:49.810]  Do you think you guys can pull this off in 35, 40 minutes?
[00:50.190 --> 00:51.330]  Probably just point out that one.
[00:51.570 --> 00:53.850]  Yeah, if we can have 40 minutes.
[00:54.270 --> 00:54.630]  That's cool.
[00:54.830 --> 00:56.470]  Yeah, I just want to try to get the closing ceremony sometime.
[00:57.050 --> 00:57.730]  We're not getting into that.
[00:58.270 --> 00:59.430]  We're not prepared.
[00:59.610 --> 01:00.010]  We don't have time.
[01:00.370 --> 01:01.610]  Do you have the channel running time out?
[01:01.690 --> 01:02.510]  Yeah, the presentation.
[01:04.070 --> 01:05.150]  That's so lame.
[01:05.430 --> 01:06.070]  What are they doing?
[01:07.130 --> 01:08.290]  Maybe retribution for 18.
[01:08.290 --> 01:08.590]  No.
[01:09.310 --> 01:10.110]  Yeah, no kidding.
[01:20.710 --> 01:21.850]  What does that represent?
[01:22.050 --> 01:22.450]  That's my smile.
[01:22.450 --> 01:24.590]  That's the number of people on the wireless.
[01:25.070 --> 01:25.730]  Oh, really?
[01:25.910 --> 01:26.050]  Yeah.
[01:26.190 --> 01:26.670]  That's great.
[01:27.810 --> 01:29.070]  You guys are geniuses.
[01:29.790 --> 01:30.570]  Markus, what am I doing?
[01:32.690 --> 01:38.210]  Well, we have pretty much here the core people involved with the network, which wouldn't have been possible without all of the volunteers.
[01:38.730 --> 01:49.330]  And we're going to do a little recap of how the network worked, what parts of it didn't work, problems we've had in the past, problems we solved this time, and problems we think we still had this conference.
[01:51.190 --> 01:56.910]  So the network team is Adam Fredo, who's the lead of all of the networking at HOPE.
[01:57.850 --> 02:03.090]  Seth Hardy was in charge of all of the network staff in the NOC this year who kept it running.
[02:03.950 --> 02:08.250]  I was in charge of the wireless, and Mike at the end there...
[02:08.250 --> 02:11.730]  I'm sorry, Mike at the end there was in charge of the public terminal cluster.
[02:14.590 --> 02:17.750]  And Adam will talk about some of the problems we've had in past years.
[02:20.290 --> 02:29.130]  So, now, what I'll talk about is really only going to be in the perspective of what happened since H2K through the Fifth HOPE.
[02:29.130 --> 02:35.090]  Although I attended HOPE in 94, I was not part of the staff, and I didn't attend beyond HOPE.
[02:35.290 --> 02:44.290]  So, my perspective only goes back to what happened around H2K, which is when I first got involved in helping with the organization of the HOPE Conference.
[02:46.210 --> 02:51.210]  So, hopefully everybody remembers what... hopefully a lot of people were here for that conference.
[02:51.670 --> 02:57.870]  And if people recall, all we had was the 18th floor, where where we're sitting was not used for speakers.
[02:57.870 --> 02:59.870]  It was used only as the network room.
[03:00.270 --> 03:05.330]  And I believe the only speaking rooms we had were... what is that?
[03:05.430 --> 03:05.790]  Pentop?
[03:06.030 --> 03:07.050]  I'm never even up here.
[03:07.130 --> 03:07.930]  I don't even remember what they're called.
[03:08.770 --> 03:09.910]  Skytop and Pentop.
[03:09.990 --> 03:10.210]  Okay.
[03:10.370 --> 03:10.750]  In Madison.
[03:10.750 --> 03:16.970]  So, those are the speaking rooms, and then the public terminal cluster was in the Madison room, if I'm not mistaken.
[03:17.510 --> 03:18.310]  Is that Madison?
[03:18.750 --> 03:18.830]  Yep.
[03:18.870 --> 03:24.910]  So, the scale of the network was significantly smaller and significantly easier.
[03:24.910 --> 03:28.250]  I believe we had everything set up in like three or four hours.
[03:28.570 --> 03:34.670]  And when it came time for collapsing everything at the end of the conference, I think I was out of here by 9 p.m.
[03:35.370 --> 03:37.530]  We did... that doesn't mean we didn't have any problems.
[03:37.530 --> 03:40.930]  For instance, wireless was a complete afterthought.
[03:41.150 --> 03:42.230]  We didn't even consider it.
[03:42.690 --> 03:48.150]  And then, Rop Gonggrijp was nice enough to go out to like RadioShack or whatever.
[03:48.450 --> 03:55.910]  Comp USA and purchase two Apple Airports and just stick them up so there were some people who could get onto the wireless network.
[03:57.190 --> 04:01.590]  Also, since the scale was so small, there was only a team of like maybe nine people.
[04:02.230 --> 04:05.210]  The core network organizers were the only people doing anything.
[04:05.370 --> 04:07.370]  There was no like real teams.
[04:07.390 --> 04:09.230]  It was just us setting up the network.
[04:11.270 --> 04:18.090]  Now, people might remember that we had a problem with Verizon where we didn't get the T1s up until about 5 p.m. on Friday, which is much after the conference start.
[04:20.730 --> 04:25.750]  But once they were up, you know, people rejoiced and we were on the network.
[04:26.090 --> 04:30.930]  So, we had a couple of hiccups on the actual router.
[04:31.090 --> 04:37.410]  We were using a stock OpenBSD machine which just could not handle the load of just a small network over here.
[04:37.650 --> 04:47.470]  So, midway at the conference, we had to rebuild OpenBSD, bump up the NMB clusters so they could actually have enough kernel memory to handle the packet switching.
[04:47.470 --> 04:56.530]  So, we had to do that a couple of times until we found the magic number which is about 16,000 which was able to handle, you know, to keep the network moving.
[04:56.970 --> 05:02.270]  That happened about Saturday and then I don't think there were any major events with the network for the remainder of the conference.
[05:02.470 --> 05:10.170]  So, I think the network for H2K was actually, you know, pretty reasonably solid, although much smaller in scale.
[05:10.850 --> 05:17.310]  So, fast-forward to 2002 when we had to set up the HOPE, the conference.
[05:18.390 --> 05:34.630]  The manuals thought that we would have enough turnout that we could have justified the expense of getting not only the 18th floor but also the second floor which, you know, had recently been transformed from a sports authority which you may see the sports equipment still lining the walls of the escalators.
[05:34.630 --> 05:38.970]  It's, you know, into what's now the convention center of the Penn Pavilion.
[05:39.950 --> 05:44.770]  And that was... we seriously underestimated how much work was going to be involved in that.
[05:45.110 --> 05:46.130]  There was a lot of problems.
[05:46.770 --> 05:52.510]  Number one, we didn't actually do a lot of preparing on the network side because we took it for granted that things went so easily last time.
[05:52.650 --> 05:54.610]  This is going to be, you know, just more of the same.
[05:54.810 --> 05:55.990]  Well, unfortunately it wasn't.
[05:56.350 --> 06:00.130]  Getting the fiber between 2 and 18 was just a nightmare.
[06:00.130 --> 06:06.550]  I think we broke 9 out of 12 of the pairs of the fiber.
[06:07.070 --> 06:09.910]  But luckily we had enough pairs that we can get 2 and 18 connected.
[06:11.130 --> 06:14.770]  There were just... the wireless network was also, again, an afterthought.
[06:14.970 --> 06:20.110]  We just... I believe the PTC guys brought up some links as we just drew them around the network.
[06:20.550 --> 06:24.290]  And I don't know how much configuration actually went into them.
[06:24.290 --> 06:29.510]  But, you know, the wireless, I think it's safe to say just... it didn't really work, unfortunately.
[06:30.390 --> 06:33.930]  A lot of people blamed the fact that we only had 2 T1s to the Internet.
[06:33.930 --> 06:39.590]  I hope after we show some of the stats that we will dispel that rumor.
[06:39.770 --> 06:45.310]  Because a lot of people were complaining this time around that we didn't have enough bandwidth to the Internet with only 2 T1s.
[06:45.750 --> 06:50.270]  Last time we had the same amount of bandwidth and people attributed the slowness to that when it was actually the wireless networks.
[06:50.270 --> 06:52.970]  Because the wired networks that we were on were, you know, pretty much fine.
[06:54.230 --> 07:00.810]  One of the major problems we had at H2K2 was just management.
[07:01.650 --> 07:05.390]  We had this mammoth undertaking of both 8th and 2nd floors.
[07:05.850 --> 07:09.710]  And we had just about the same number of volunteers as last time.
[07:09.930 --> 07:13.190]  And I tried to manage both the volunteers and the network.
[07:13.370 --> 07:15.450]  And that just... that was just a complete failure.
[07:16.470 --> 07:24.410]  When it came down to the time of the conference, I had made note of about 20 people who were volunteering to handle the NOC staff.
[07:24.790 --> 07:27.690]  And I think I didn't get any of them.
[07:28.110 --> 07:32.390]  And the only people I actually had were the people who helped out the conference the last time around.
[07:32.790 --> 07:34.570]  So I didn't actually sleep.
[07:34.710 --> 07:37.550]  I was at the conference from maybe 7 p.m. till 7 a.m. for the three days.
[07:39.610 --> 07:41.790]  And I had one person here taking the 7 a.m. to 7 p.m. shifts.
[07:42.650 --> 07:44.510]  And it was just way too much.
[07:44.970 --> 07:46.310]  I didn't get to see any of the conference.
[07:46.530 --> 07:48.510]  I just hung out at the NOC making sure everything was running.
[07:50.770 --> 07:54.410]  So obviously there were a lot of problems we had in planning everything.
[07:54.670 --> 07:58.090]  Oh, luckily the only problem we didn't have was Verizon.
[07:58.250 --> 08:00.390]  They brought in the T1s a month early.
[08:01.310 --> 08:04.450]  And the Internet was up before I even got to the conference.
[08:05.390 --> 08:07.530]  NOC.NET came over and set up the T1s.
[08:07.590 --> 08:08.970]  And everything was working when I got there.
[08:09.950 --> 08:12.270]  But everything else was just a...
[08:12.730 --> 08:15.510]  It was almost a miserable experience.
[08:15.510 --> 08:16.350]  But it was a lot of fun.
[08:16.430 --> 08:17.690]  I met a lot of good people.
[08:17.870 --> 08:21.310]  And it was fun working with people under such an enormous amount of stress.
[08:21.490 --> 08:24.550]  But the conference itself just wasn't a lot of fun for us.
[08:24.550 --> 08:31.350]  So this time around, the network team and I, we just planned out...
[08:31.350 --> 08:36.650]  We actually did some foreplanning and made sure we understood what we were getting into before we got into it.
[08:36.710 --> 08:45.650]  And I think, as you'll see after we finish speaking, the network was a very good success, at least in my opinion, and hopefully a lot of your opinions.
[08:45.910 --> 08:52.130]  So I guess we should just start talking about the planning that went involved in this year's conference.
[08:53.890 --> 08:58.610]  So this time around, one of the things that I made sure of...
[08:58.610 --> 09:08.170]  And unfortunately for Seth, he had the responsibility for handling all of the NOC staff managing the volunteers.
[09:08.430 --> 09:12.150]  And I think he can attest to what a difficult job that was.
[09:12.430 --> 09:16.590]  We did have a lot of really good people in the NOC this time.
[09:16.770 --> 09:18.750]  And that's really a testament to Seth.
[09:18.950 --> 09:22.570]  He made sure that we got enough people and enough people could actually trust.
[09:22.790 --> 09:34.750]  We didn't have any jerks trying to get into the NOC and start tinkering around with stuff to sabotage what we had, which is my biggest fear because I had such a hands-off approach to the NOC staff.
[09:34.890 --> 09:39.870]  But he did a really outstanding job getting some good people, some very good, very smart people.
[09:41.030 --> 09:44.330]  And I guess Seth will talk more about that when I hand it over to him.
[09:44.330 --> 09:55.430]  And I guess on the core network side, we actually set up the machines ahead of time as opposed to setting up the machines at the conference.
[09:55.650 --> 09:57.890]  So we had pretty much everything ready to go.
[09:58.350 --> 10:06.790]  When we started setting up the network at 5pm, actually at 4pm on Thursday, the network was actually up by 5pm.
[10:06.930 --> 10:10.130]  All we had to do is plug everything in, turn them on, and it was ready to go.
[10:10.130 --> 10:13.030]  So, unfortunately, we didn't have the T1s in place.
[10:13.210 --> 10:17.230]  But, you know, that's another story we'll get into later on.
[10:18.750 --> 10:19.630]  I think...
[10:19.630 --> 10:25.250]  So, I guess I'll hand it over to Seth now to tell what he might have done for the NOC staff.
[10:25.830 --> 10:31.510]  So, it all started when there was one of the Fifth HOPE volunteer meetings in the city.
[10:32.030 --> 10:37.390]  And there was talk about how, in previous years, there weren't a ton of people on NOC staff.
[10:37.810 --> 10:42.770]  And I made the mistake of opening my mouth and going, I can get you coverage for the entire weekend.
[10:43.070 --> 10:46.170]  And right after I said it, I'm like, oh, I'm screwed.
[10:46.290 --> 10:47.150]  I just screwed myself.
[10:48.230 --> 10:51.810]  And immediately, people were like, if you can do that, do it, you're in charge.
[10:51.890 --> 10:53.130]  And I'm like, oh no, what have I done?
[10:53.790 --> 11:04.090]  I got maybe 35, 40 people for NOC staff over the next few months, mostly because I knew about half of them were going to be scumbags and not show up.
[11:04.810 --> 11:06.410]  It wasn't actually that bad.
[11:06.630 --> 11:08.550]  A lot of people showed up.
[11:08.850 --> 11:10.790]  There were people there at all times.
[11:11.630 --> 11:22.950]  I think we had enough really good people there who weren't just showing up for their shifts, but they actually wanted to help out and assist in every way that they could because they enjoyed that sort of thing.
[11:23.110 --> 11:25.730]  They wanted to see the network work properly.
[11:25.730 --> 11:27.770]  They wanted to help everybody else out.
[11:27.770 --> 11:34.690]  So we had a lot of people who were just hanging out in the NOC for a good portion of the conference because it's what they wanted to do.
[11:35.250 --> 11:37.490]  So we had a whole bunch of people there helping out.
[11:38.170 --> 11:43.570]  When people who weren't as committed didn't show, it wasn't as much of a problem.
[11:44.390 --> 11:46.250]  But it turned out really, really well.
[11:46.390 --> 11:48.090]  It was a lot better than I was actually expecting.
[11:48.510 --> 11:51.830]  I was sort of worried that I was going to get the 4 a.m. phone calls saying, Hey, nobody showed up for their shift.
[11:55.150 --> 11:56.230]  You have to come and cover.
[11:57.050 --> 11:58.850]  So that didn't really happen.
[11:59.210 --> 12:00.210]  Everything went really well.
[12:00.690 --> 12:09.670]  And really, it's the volunteers who signed up and then spent most of their time at the conference in the NOC doing work.
[12:09.910 --> 12:11.430]  I mean, they paid to get in.
[12:11.890 --> 12:28.030]  And rather than go do all the things that a badge that you paid for would give you the privilege to do, rather than do that, they decided that they would put in their time and their effort and their energy and their money even to getting everything up and running and keeping it running,
[12:28.430 --> 12:32.450]  even if it meant missing out on talks, even if it meant missing out on sleep.
[12:32.790 --> 12:35.930]  A lot of us really didn't sleep much, as you can probably tell.
[12:38.070 --> 12:45.890]  But these people are really the ones, you know, who deserve the credit for the staffing stuff, not me, because I just got them to show up.
[12:46.190 --> 12:48.530]  And the people who wanted to be there were there.
[12:48.670 --> 12:50.350]  And so it's really up to them.
[13:00.250 --> 13:03.610]  I guess that's really all I have to say about the staff unless there are any questions.
[13:04.650 --> 13:05.010]  But...
[13:05.010 --> 13:06.650]  We'll do the Q&A part at the end.
[13:06.970 --> 13:07.670]  We're actually...
[13:08.450 --> 13:14.130]  Unfortunately, we banged out this panel about, I guess, 45 minutes ago.
[13:14.490 --> 13:15.290]  And Seth wasn't...
[13:15.650 --> 13:17.210]  Seth and Mike, unfortunately, weren't there.
[13:17.370 --> 13:18.750]  So we actually have a lot more than that.
[13:18.870 --> 13:20.130]  We're gonna actually just talk about...
[13:20.130 --> 13:25.750]  After we talk about the planning, we'll talk about how we implemented it, shows you some network statistics, bandwidth and stuff.
[13:26.270 --> 13:28.390]  And then we'll talk about what the problems were.
[13:28.630 --> 13:29.730]  And then we'll go to Q&A.
[13:30.350 --> 13:33.830]  So, Mike, is there anything you want to talk about how the wireless was planned?
[13:34.130 --> 13:34.370]  Right.
[13:35.370 --> 13:41.390]  In the past, as Adam said, the wireless was mostly consumer equipment, which is pretty easy to make fall down.
[13:42.050 --> 13:44.390]  This time, we tried to plan it so that...
[13:45.430 --> 13:53.530]  Well, basically, there's no way you can keep people from breaking the wireless because it's a broadcast carrier and someone's gonna transmit and make it fall down.
[13:53.810 --> 14:00.190]  All we can do is make it hard for them and make it so that the system will recover itself as much as possible from someone doing that.
[14:00.490 --> 14:03.290]  And for a large part, we were fairly successful with that this year.
[14:04.110 --> 14:08.370]  A couple people decided to screw with the network who were moderately successful at it, unfortunately.
[14:09.050 --> 14:10.610]  But for the most part, it stayed up.
[14:10.610 --> 14:19.770]  We used custom hardware that we were borrowing from the Mesh AX people in Boston, who are part of the RoofNet community networks.
[14:20.770 --> 14:33.750]  And we used that, some custom software to manage it, host AP under Linux to run the access points themselves, high-powered radios, and a lot of coverage to try to defeat people messing with the network.
[14:34.250 --> 14:39.450]  If you look up there, actually, on the curtains, I don't know how many noticed them, actually during the con.
[14:39.650 --> 14:41.750]  That's where the access points live for this room.
[14:45.010 --> 14:46.970]  The planning was about two months in.
[14:47.230 --> 14:52.690]  We got a walkthrough early and did some radio testing and actually mapped out the networks this time.
[14:54.010 --> 14:58.270]  And I'll be releasing the software that controlled the networks after the conference.
[14:58.450 --> 15:01.530]  But I wasn't gonna give everyone a chance to find my coding bugs beforehand.
[15:06.930 --> 15:09.930]  Someone had a question, but we'll take it at the end, I guess.
[15:10.090 --> 15:12.430]  I couldn't really hear you and we'll get people at the microphone after.
[15:13.890 --> 15:17.030]  So, turn it over to Mike to talk about the PTC a bit.
[15:17.490 --> 15:21.930]  Yeah, I'd like to start out with, can I get a show of hands how many people here have actually used one of the public terminals?
[15:22.730 --> 15:23.170]  Yeah!
[15:24.010 --> 15:24.830]  Alright, great.
[15:25.470 --> 15:31.250]  The public terminal cluster is a little bit weird because it's not really part of the core 2600 experience.
[15:32.230 --> 15:37.010]  The hope organizers provide a network and they expect you to bring their own computer.
[15:37.190 --> 15:39.750]  And I always thought, well, you know, what if you don't have your own computer?
[15:39.990 --> 15:41.590]  And that's sort of where we step in.
[15:41.810 --> 15:47.250]  And one of the problems we have is since we're not really part of the core, we don't get a budget.
[15:47.410 --> 15:49.790]  So everything they have to do has to be very, very cheap.
[15:50.310 --> 15:55.930]  This year, we got about 50 or so 46's from Dragorn.
[15:56.230 --> 15:59.810]  We got a lot of free monitors and that's what we had to work with.
[16:00.350 --> 16:05.910]  We tried to do this two years ago and it was really a complete disaster.
[16:06.470 --> 16:10.530]  And one of the main problems we had was that we thought, well, there's going to be all these people.
[16:10.650 --> 16:12.610]  We need, you know, 70 to 100 machines.
[16:13.090 --> 16:18.990]  And just bringing all of those to the conference and trying to get them up and running really consumed most of our time.
[16:22.710 --> 16:25.170]  This year we decided we're just not going to do that.
[16:25.330 --> 16:30.990]  We have about 16 to 20 things running on the second floor and a couple more up here on the first floor.
[16:31.150 --> 16:32.670]  And really that was enough.
[16:32.850 --> 16:34.690]  A lot more people do have laptops now.
[16:34.690 --> 16:41.850]  But at the same time, I did see pretty much people on the public terms almost 24-7 as well.
[16:42.870 --> 16:48.590]  We got around some of the problems with the network actually on Friday for a good bit of the afternoon.
[16:48.850 --> 16:50.530]  The core T1s were down.
[16:50.550 --> 16:54.550]  So we found someone who donated us a little bit of wireless to get online.
[16:54.550 --> 16:56.450]  They don't know that they donated it.
[16:56.650 --> 16:57.770]  But I thank them anyway.
[16:58.650 --> 17:02.250]  So we were actually online on the public cluster when everybody else was down.
[17:02.650 --> 17:05.570]  Which is something we've never been able to pull off before.
[17:06.630 --> 17:10.290]  We really have a lot more problems than the network itself does.
[17:10.510 --> 17:12.890]  Because the network only has to provide network connectivity.
[17:13.350 --> 17:15.070]  We have all of the problems that they do.
[17:15.070 --> 17:18.390]  And then on top of that, we have servers that are being attacked.
[17:19.390 --> 17:28.550]  The other co-conspirator, Mark, who's somewhere in the audience there, said when he walked around, pretty much every time he saw at least one person trying to do something to our server.
[17:28.990 --> 17:32.710]  And in the face of that, I'm really surprised how well it's worked out.
[17:33.450 --> 17:36.630]  We plan to have network connectivity on all three floors.
[17:37.330 --> 17:39.910]  I wasn't really sure that we'd be able to pull it off.
[17:39.910 --> 17:44.930]  But I woke up this morning and there are usable terminals on all three floors.
[17:45.310 --> 17:46.790]  I was pretty happy about that.
[17:48.330 --> 17:52.270]  Another problem that I had in planning is I myself live in Montana.
[17:52.890 --> 17:58.270]  Years ago when HOPE first started, I lived in Pittsburgh and so did most of the other people involved.
[17:58.390 --> 18:02.030]  Now we're spread out between Montana and Seattle and so on.
[18:02.710 --> 18:08.750]  And two of the people that helped out a lot, Cy, Pete, and Adam, I didn't even meet until Thursday.
[18:08.750 --> 18:12.990]  But we were able to collaborate over the Internet to get everything working.
[18:13.250 --> 18:14.170]  Pete brought some things.
[18:14.270 --> 18:15.510]  I brought some things.
[18:15.810 --> 18:19.870]  And when we got here, it all kind of came together really well.
[18:21.350 --> 18:27.230]  Because we only have 486s to work with, obviously you're not going to be able to run a modern web browser directly on them.
[18:27.390 --> 18:32.390]  So I recycled a concept that was popular in the mid-90s, which is the Thin Client.
[18:33.290 --> 18:37.530]  The terminals themselves, all they do is run the graphics in the windowing system.
[18:37.530 --> 18:39.630]  They don't run any of the applications you see.
[18:39.810 --> 18:43.150]  All those were run on one or two servers during the con.
[18:43.750 --> 18:45.750]  And that worked out pretty well.
[18:45.910 --> 18:49.610]  We were afraid that our servers would get overloaded and we need a whole lot of them.
[18:49.870 --> 18:55.410]  But we actually managed to pull it off with really only one machine and that was only an Athlon 1900.
[18:56.910 --> 18:58.910]  That's why the DCC did not work.
[18:59.110 --> 18:59.330]  Okay.
[19:00.050 --> 19:01.590]  Yeah, because we're using that.
[19:01.690 --> 19:03.850]  It was remote, so it was in the wrong place.
[19:04.110 --> 19:04.510]  Yeah.
[19:04.970 --> 19:05.370]  Yeah.
[19:06.710 --> 19:15.410]  Another big problem we had at H2K2 and we had this time too is that, of course, if you have an Ethernet network, people are going to go in and screw with it.
[19:15.530 --> 19:19.570]  They're going to announce themselves as a default route and do all kinds of horrible things.
[19:19.570 --> 19:22.970]  So we got around that this year by basically not having Ethernet.
[19:23.330 --> 19:27.390]  The machines up here are Ethernet, but the ones on one and two are all token ring.
[19:27.970 --> 19:30.210]  And only one person brought a token ring device.
[19:30.210 --> 19:31.670]  I actually thought it was going to be zero.
[19:32.090 --> 19:37.610]  A lot of people tried to plug into it and, you know, couldn't figure out why they weren't getting Ethernet signals on our token ring.
[19:38.830 --> 19:44.390]  We put Ethernet hubs on top so that you could still bring your laptop and sit next to your friends.
[19:44.390 --> 19:49.690]  We also made sure there was space on the tables to put your laptops and that went pretty well.
[19:50.290 --> 19:53.690]  A big problem still was people yanking cables out all the time.
[19:53.890 --> 19:56.090]  That was probably the main reason that machines were down.
[19:56.810 --> 20:02.230]  And short of coming here with steel boxes to put everything in, I don't know if there's any way to prevent that.
[20:04.650 --> 20:06.470]  I'm really happy about the Ataris.
[20:06.730 --> 20:11.550]  I think maybe next time I'm just going to bring a couple of Ataris because they're so gratifying.
[20:11.550 --> 20:15.470]  We set them up on Thursday and pretty much from that point on people were playing Atari.
[20:16.010 --> 20:17.130]  That was really nice.
[20:23.790 --> 20:27.230]  Yeah, I'm not... I think that's pretty much all I have on planning.
[20:28.390 --> 20:28.750]  All right.
[20:31.450 --> 20:39.430]  So, I guess we actually covered a lot of the implementation during our discussions about the planning.
[20:39.430 --> 20:43.210]  But I'll just go into a little more detail about how the layout was.
[20:44.630 --> 20:47.930]  Mike, can you switch to that network map that I couldn't get into the thing?
[20:53.410 --> 20:55.430]  Yeah, watch the Linux guy try to use Windows.
[20:55.630 --> 20:56.250]  Isn't this fun?
[21:00.410 --> 21:01.990]  Try one of the Internet Explorers.
[21:02.270 --> 21:03.590]  One of the Internet Explorers.
[21:04.330 --> 21:06.530]  Okay, now make that window big.
[21:08.470 --> 21:09.210]  All right.
[21:09.370 --> 21:15.090]  So, that's a topology map that I drew up in Visio about the network layout.
[21:15.950 --> 21:17.350]  It's pretty straightforward.
[21:17.770 --> 21:21.810]  We organized the groups of machines, you know, into classes.
[21:22.570 --> 21:24.650]  The public terminal cluster was one class.
[21:24.890 --> 21:27.790]  The bring your own computer area was another class.
[21:28.650 --> 21:29.450]  There was...
[21:29.450 --> 21:30.250]  Let's see.
[21:30.350 --> 21:31.950]  The PTC was in their own area.
[21:32.210 --> 21:32.750]  It was...
[21:32.750 --> 21:35.430]  I keep saying class, but I keep meaning to use the word VLAN.
[21:36.750 --> 21:37.330]  Let's see.
[21:37.450 --> 21:40.950]  We had a knock VLAN for all the people who are actually working on the network.
[21:41.510 --> 21:50.490]  A VLAN for the servers that were, you know, running some of the services like DNS, DHCP, Apache, MySQL for some of the internal things that we were running.
[21:50.490 --> 21:52.450]  And let's see.
[21:52.610 --> 21:55.690]  And then there was a management VLAN where all the AV equipment is.
[21:56.550 --> 22:03.610]  And so, you know, the switches for, you know, network management and pulling down, you know, SNMP data for the MFTG graphs.
[22:03.850 --> 22:10.490]  So, all of these were, you know, connected to a couple of foundry switches.
[22:11.910 --> 22:18.410]  I forget which model that Adam Rothschild brought, which was also handling our 2 to 18 fiber trunk.
[22:18.410 --> 22:27.630]  And, um, the machine that was handling all the internal routing was a, uh, NetBSD machine that, uh, was built by, uh, Miles and myself.
[22:28.170 --> 22:30.930]  Um, Miles Norton who worked on, on the Knox staff.
[22:31.130 --> 22:35.170]  Uh, we used, uh, NetBSD because we, uh, you know, were comfortable with it.
[22:35.290 --> 22:37.670]  We, uh, we know its, uh, ability to handle VLANs.
[22:37.750 --> 22:41.610]  We also had some, um, some, uh, protection built into it.
[22:41.690 --> 22:44.410]  Uh, I believe, uh, some of the subnets were ARP-less subnets.
[22:44.590 --> 22:47.550]  So, uh, to try to avoid some of the, uh, network, uh, tricks.
[22:47.550 --> 22:51.990]  I guess we can get him up here later if he's around to, uh, talk about what he might have done there.
[22:52.770 --> 22:58.690]  Um, and then, uh, you know, NAC, uh, Network Access Corp provided the, uh, 2T1s to the Internet.
[22:59.390 --> 23:03.050]  And, uh, that's really, you know, uh, how it was laid out.
[23:03.310 --> 23:06.390]  Um, I was running IPF on the router.
[23:06.390 --> 23:10.250]  And I know some people were complaining, oh, the network is a little too restrictive.
[23:10.510 --> 23:12.150]  You're, you guys are fascists and stuff.
[23:12.730 --> 23:16.530]  Um, well, you know, uh, actually the network wasn't as restricted as you think.
[23:16.530 --> 23:20.550]  Uh, we did allow everything out from the, uh, internal networks.
[23:20.550 --> 23:24.590]  What we didn't allow out was anything from, you know, private address space.
[23:24.710 --> 23:27.410]  Because we were using private address space, we needed to use it.
[23:27.730 --> 23:34.730]  We needed to, um, you know, as much rooms we wanted you guys to have access to the Internet, we needed to make sure that things kept running.
[23:34.730 --> 23:39.470]  So, uh, we, we did protect, uh, protect the infrastructure from, you know, from that stuff.
[23:39.630 --> 23:45.030]  But, you know, we didn't restrict you guys from, you know, scan, port scanning the Internet, doing anything you wanted to do on the Internet.
[23:45.210 --> 23:48.930]  Just as long as it was traced back to original, you know, and, uh, correct IP address.
[23:49.090 --> 23:52.910]  That's the only restriction that there was on the, uh, you know, uh, on the routing.
[23:53.350 --> 23:57.030]  So, um, I think that's about it from my side.
[23:57.030 --> 24:03.470]  Um, actually the only other things, uh, we, we ran fiber, you know, down the stairwell like we did last time.
[24:03.650 --> 24:06.610]  Uh, we had a problem where, uh, we thought we miscalculated the length.
[24:07.110 --> 24:11.130]  Um, somebody thought it was a 1250 foot spool and it was actually a 250 foot spool.
[24:11.170 --> 24:12.750]  So we got cut off halfway down.
[24:13.010 --> 24:16.930]  Um, and this was about, I guess, uh, midnight when we found this out.
[24:17.110 --> 24:22.590]  So now we're scrambling to get some, uh, cabling for, uh, you know, some way to hook up our other spool of fiber.
[24:22.810 --> 24:26.770]  So somebody ran to his office, got these, uh, SC to SC couplers.
[24:26.770 --> 24:30.490]  Uh, we connected them together, taped it up, ran the cable the rest of the way.
[24:30.850 --> 24:33.230]  And voila, we had two to 18 connected.
[24:33.730 --> 24:35.650]  Um, let's see.
[24:35.810 --> 24:38.190]  Uh, I guess the, the network up here is pretty straightforward.
[24:38.190 --> 24:42.590]  There's just a switch handling, uh, the AV equipment sets around and the wireless devices.
[24:42.770 --> 24:45.570]  There's not, nothing, uh, pretty special about the, uh, network up here.
[24:46.010 --> 24:49.230]  Um, I guess that's it from my side, if you want to hand it out.
[24:49.330 --> 24:52.770]  Seth, is there anything in particular about, uh, how you handle the Knox staff that you want to talk about?
[24:53.370 --> 24:56.750]  Um, mostly it just involved a lot of yelling at people when they didn't show up.
[24:56.750 --> 25:02.570]  Um, people who are dedicated were there all the time and they were absolutely excellent.
[25:02.590 --> 25:06.450]  People who weren't, were invisible because they weren't there.
[25:06.750 --> 25:12.670]  But we did enough planning in advance and we got enough people who were trustworthy that it wasn't an issue.
[25:13.210 --> 25:18.490]  Um, originally wanted to have it be three hour shifts.
[25:18.490 --> 25:22.030]  Um, it ended up being four hour shifts.
[25:22.650 --> 25:25.310]  Um, everybody was given two shifts.
[25:25.550 --> 25:28.330]  Um, they weren't technically allowed to have more than two.
[25:28.510 --> 25:32.530]  Uh, so that people wouldn't burn out and not show up when they were supposed to be there.
[25:32.530 --> 25:39.330]  But the people who were really involved spent a lot more than two shifts worth in the Knox.
[25:39.610 --> 25:41.010]  And that's very, very appreciated.
[25:41.850 --> 25:45.350]  Um, we had three people officially on duty at any given time.
[25:45.990 --> 25:49.850]  Um, there were times when we had 15 people plus in the Knox.
[25:50.090 --> 25:52.050]  Um, actual Knox staff.
[25:52.050 --> 25:57.550]  Um, we got some badges for the Knox staff and actively kept everybody else out.
[25:57.830 --> 25:59.810]  So that people wouldn't get distracted.
[26:00.190 --> 26:06.150]  So that nobody would wander in and steal equipment or break things or touch things that they weren't supposed to touch.
[26:06.710 --> 26:08.230]  And that worked out really well.
[26:08.730 --> 26:13.390]  Um, our physical location on the second floor was kind of hard to secure.
[26:13.790 --> 26:19.890]  So we ended up grabbing things like curtains and tables and barricading the back and trying to keep people out.
[26:19.890 --> 26:24.750]  And I was actually expecting a lot more people to try to sneak their way in and, you know, get into the Knox.
[26:24.990 --> 26:27.470]  Because I'm not allowed to be in there, so I have to try.
[26:28.290 --> 26:29.990]  Uh, we did have some of that.
[26:30.130 --> 26:34.270]  But for the most part it was actually pretty good, pretty isolated from everybody else.
[26:34.430 --> 26:35.950]  People were actually able to do work.
[26:37.550 --> 26:40.150]  Um, we had some space in the back.
[26:40.370 --> 26:41.410]  Uh, I brought a tent.
[26:42.030 --> 26:44.630]  There's a little pitch tent in the back that people slept in.
[26:45.230 --> 26:47.770]  Uh, because they didn't want to leave.
[26:47.770 --> 26:49.270]  They wanted to stay around the Knox.
[26:49.450 --> 26:50.810]  So they slept in a tent in the back.
[26:51.450 --> 26:53.470]  Uh, very dedicated people.
[26:54.090 --> 26:56.150]  Um, and it went well.
[26:56.310 --> 26:57.670]  It went a lot better than I was expecting.
[26:57.710 --> 26:59.570]  So I'm very thankful for that.
[27:00.890 --> 27:02.890]  That's really all I have to say about it.
[27:03.330 --> 27:04.610]  Is, is Miles available?
[27:05.650 --> 27:06.890]  I know where Miles is.
[27:07.610 --> 27:08.590]  Is Miles out there?
[27:08.730 --> 27:10.590]  Do you want to come up and talk about your ARPLIST stuff?
[27:10.590 --> 27:23.230]  Uh, one thing that we did before even getting here was there was a lot of volunteer work, um, in terms of setting the network up and the design and everything going on at our apartment.
[27:23.850 --> 27:27.840]  And one of my roommates, Miles, did a lot of work on the ARPLIST stuff.
[27:27.840 --> 27:29.860]  So he's on his way up right now.
[27:30.100 --> 27:37.940]  But I definitely think that he should have the chance to talk about it if you'd like, if you'd like, because he put in a lot of work into it.
[27:38.300 --> 27:40.760]  And, yeah, so...
[27:40.760 --> 27:41.600]  You can come up here.
[27:45.810 --> 27:47.430]  If you think it's useful, I can do it.
[27:51.230 --> 27:52.550]  Oh, well, that one is up.
[27:52.710 --> 27:55.410]  I'll, um, you can talk to me afterwards, uh, for the URL.
[27:57.950 --> 28:01.830]  Um, yeah, the ARPLIST stuff didn't really get a good test.
[28:02.010 --> 28:03.730]  I don't think anybody hammered on it.
[28:03.830 --> 28:13.170]  I didn't, uh, provably reject any attacks, you know, like, compared to the pages and pages of logs that Dragorn has for the wireless attacks.
[28:13.170 --> 28:18.630]  Um, but, uh, the idea was that the ARP spoofing is fairly simple.
[28:18.770 --> 28:21.230]  People would either ARP spoof the gateway or the client.
[28:21.710 --> 28:26.010]  And the stuff that I did was invisible to ordinary users.
[28:26.250 --> 28:38.850]  Um, and it only protected against people trying to ARP spoof the client and convince the gateway into sending packets to the attacker's machine rather than to the actual client.
[28:39.270 --> 28:50.750]  Um, in order to prevent the more common attack of ARP spoofing the gateway, the individual user of each machine would have to statically configure the MAC address of the gateway into his machine.
[28:51.030 --> 28:54.250]  And, uh, I don't know anybody who did that.
[28:54.350 --> 28:56.470]  I did that to my machines when I started getting attacked.
[28:56.890 --> 28:59.390]  Uh, but I didn't announce it, and that was kind of a weakness.
[28:59.390 --> 29:04.330]  But, um, yeah, it was, uh, uh, largely transparent.
[29:08.220 --> 29:08.800]  All right.
[29:09.260 --> 29:09.440]  Thanks.
[29:10.560 --> 29:23.620]  Well, we mentioned some of the wireless implementation, uh, briefly before, but, uh, the actual hardware was a Cyrix Media GX board, which hasn't been made for about two years, but the guys in Boston managed to get their hands on a couple dozen of them.
[29:23.620 --> 29:39.060]  Um, and it's a neat little ATX board, uh, that we crammed in a PrimeStar satellite distribution box, little gray box, uh, downclocked it to prevent some overheating, uh, slapped a compact flash hard drive in it, and two 200 milliwatt radios.
[29:39.880 --> 29:47.600]  Um, the software itself was designed so that they would all report to a central controller that's down in the NOC, which would log all the attacks that we were seeing.
[29:47.840 --> 29:50.060]  Uh, it would automatically configure all the channels.
[29:50.060 --> 29:54.820]  So if an AP went away, it would reconfigure the rest of them to take advantage of the available channels.
[29:55.440 --> 30:00.580]  It ran Kismet on some of the access points to watch for people attacking the network.
[30:01.020 --> 30:04.680]  And it could also do, uh, some active counterattacking.
[30:05.100 --> 30:15.820]  So, if someone brought up an access point named the same as ours trying to spoof ours, it would first flood all the users on the fake access point off so that they've reattached to the real access point.
[30:16.160 --> 30:18.080]  Uh, sort of a reverse man in the middle.
[30:18.080 --> 30:24.640]  And then it would do the 802.11 equivalent of a sin flood against the fake access point until it fell down.
[30:26.020 --> 30:38.800]  Um, we eventually turned that off because there were so many people attacking the access points that the airwaves were just so clogged that it paid off more to let people get stuck on the spoofed ones sometimes than to contribute to the congestion in the air.
[30:39.040 --> 30:47.900]  At one point, we were running 1,600 to 1,700 packets a second on one channel in the wireless, which is absolutely insane.
[30:48.780 --> 30:49.280]  Um...
[30:49.280 --> 30:50.700]  Don't ever hope to have kids.
[30:51.040 --> 30:51.380]  Sorry.
[30:54.620 --> 30:59.280]  After having all the 200 milliwatt radios in one room at the same time, I've already given up on that hope.
[31:00.480 --> 31:01.200]  Blame him.
[31:01.780 --> 31:03.600]  I thought you were wearing cold all day.
[31:05.140 --> 31:08.940]  Um, so that was pretty much how the wireless worked.
[31:09.300 --> 31:11.180]  Uh, some APs flapped around a bit.
[31:11.600 --> 31:13.240]  A lot of people attacked them.
[31:13.540 --> 31:17.460]  Uh, a big, you know, boot to the head to the people who messed it up for everyone else.
[31:17.660 --> 31:20.800]  But for the most part, it seemed to be holding up under the attacks.
[31:21.080 --> 31:23.220]  Uh, the implementation worked out pretty well.
[31:23.220 --> 31:29.200]  And once I've genericized the software a bit, uh, like I said, it'll be released probably off the Kismet site.
[31:29.400 --> 31:35.180]  So, if anyone's interested in how that worked, uh, I'll be writing up a readme and posting that probably in the next week or two.
[31:36.700 --> 31:37.280]  All right.
[31:37.360 --> 31:38.320]  Mike, want to talk about the Linux?
[31:38.720 --> 31:41.880]  Yeah, I kind of covered a lot of my implementation, I guess, earlier.
[31:42.160 --> 31:53.000]  Um, the one extra piece is what's actually running on the, the 46's themselves is a Linux distribution I made two years ago for, uh, H2K2.
[31:53.200 --> 31:54.140]  It's very small.
[31:54.460 --> 31:56.360]  Uh, the image itself is 128 megs.
[31:56.440 --> 32:00.560]  There's only about 70 megs of stuff in the, uh, file system itself.
[32:01.120 --> 32:07.780]  Um, and, uh, the only account is root, so if you get into it at all, um, you know, that's what's gonna be there.
[32:08.280 --> 32:09.200]  Uh, it held up pretty well.
[32:09.300 --> 32:14.820]  There's a really glaring, uh, error that I had, which is that I left the, uh, bootloader command line open.
[32:14.940 --> 32:16.540]  People were able to get in through that.
[32:16.940 --> 32:19.100]  Uh, other than that, I think it held up pretty well.
[32:19.100 --> 32:23.200]  Um, our servers were just stock FreeBSD and a Linux one.
[32:23.380 --> 32:26.260]  And, uh, as far as I know, no one got into the FreeBSD one.
[32:26.700 --> 32:36.480]  Um, so even though that Linux software is two years old, it's been battle-tested through two hope conventions, and I'm pretty confident that's actually quite secure at this point.
[32:39.000 --> 32:40.540]  Yeah, that's about it.
[32:40.860 --> 32:43.920]  All right, um, I guess, uh, that pretty much covered everything.
[32:44.240 --> 32:46.860]  Um, now, I guess we skipped something.
[32:46.860 --> 32:48.920]  I thought we had another slide there to explain that.
[32:49.060 --> 32:51.060]  This, now we're up to the network statistics area.
[32:51.640 --> 32:53.460]  So, Mike can explain this.
[32:53.780 --> 32:55.240]  Yeah, slides got reordered slightly.
[32:55.740 --> 33:02.000]  Um, this is the number of users connecting to the access points on each of the zones.
[33:02.180 --> 33:06.780]  There's two primary zones on the 18th floor, and one zone on the second floor.
[33:07.440 --> 33:12.500]  So, uh, this is pretty much over the duration of the conference, so it got a lot of use.
[33:13.020 --> 33:24.720]  Uh, this graph doesn't reflect it as well as the graphs we'll have in a minute, which are the number of addresses we handed out to unique, uh, MAC addresses over DHCP, which really reflect the, uh, number of users we were seeing.
[33:26.320 --> 33:27.120]  Uh, okay.
[33:27.300 --> 33:29.600]  Well, this is, uh, our MRTG graphs.
[33:29.760 --> 33:31.940]  I think you could see, uh, you could spot a couple of things.
[33:31.940 --> 33:40.720]  Actually, unfortunately, it's three days later, so we can't, uh, notice the time where the T1, uh, T1s were unavailable, uh, now that it's compressed, uh, you know, into the weekly graph.
[33:41.180 --> 33:46.380]  But, um, as you see, we, you know, pretty much did peak at, uh, you know, the full T1 bandwidth.
[33:46.840 --> 33:51.380]  Uh, but, um, this should hopefully dispel some of the, uh, rumors.
[33:51.600 --> 33:58.940]  Uh, I believe, um, I'm sure everybody felt that the T1s were, um, your Internet access was a little slow.
[33:58.940 --> 34:07.820]  Uh, I don't think it was slow to the point of being unusable because I was actually SSH'd out to my shell box and SSH'd back in to do some of the management.
[34:08.060 --> 34:15.680]  And I had, and I was, uh, doing interactive typing, and, uh, once in a while we'd get a little bit choppy, but for the most part it was pretty, you know, uh, it was pretty damn fast.
[34:15.880 --> 34:20.680]  So, um, that was, you know, during, uh, our, you know, saturation of the T1.
[34:21.000 --> 34:26.480]  So, um, I'm pretty confident that, um, you know, the two T1s of bandwidth were plenty.
[34:26.480 --> 34:33.400]  We didn't need, you know, 100 megabits in the Internet like some people thought, you know, uh, we needed in order to handle all of these wireless users.
[34:33.860 --> 34:36.980]  Uh, you can skip to the next one, which looks exactly the same.
[34:37.340 --> 34:40.200]  Uh, the, uh, there was two T1s going to the same router.
[34:40.340 --> 34:41.520]  They were obviously load balanced.
[34:41.520 --> 34:46.520]  So, obviously, the, uh, the picture should look the same except for, uh, if you looked three days before.
[34:46.900 --> 34:56.140]  Yeah, I think, um, okay, this is actually a better map because that very first, uh, spike, uh, I believe, if I'm not mistaken, that's when our T1s were up.
[34:56.260 --> 35:01.340]  And then when the spike, uh, you know, where it's blank, that's when the T1s were down.
[35:01.640 --> 35:03.080]  And then the T1s came back up.
[35:03.240 --> 35:04.580]  And then the T1s went down.
[35:04.740 --> 35:06.280]  And then the T1s were up for the rest of the conference.
[35:06.280 --> 35:10.200]  So that's pretty much what the, what, uh, some of the problems I'll explain in a few minutes.
[35:10.440 --> 35:11.680]  Uh, I guess we don't have too much time.
[35:11.760 --> 35:14.240]  But hopefully we'll get into what happened with Verizon, the Verizon ordeal.
[35:14.760 --> 35:15.660]  Uh, go to the next slide.
[35:16.120 --> 35:16.440]  All right.
[35:16.580 --> 35:18.300]  So, oh, wow, you can't really read that.
[35:18.520 --> 35:21.220]  But, uh, can you zoom that in any chance, anyway?
[35:21.500 --> 35:25.100]  Anyway, this is a breakout of the, um, the stats per day.
[35:25.400 --> 35:29.740]  Seth, if you can give me, uh, just the rundown of, um, uh, for each day.
[35:29.960 --> 35:32.820]  What, what the wireless users were compared to the wired users.
[35:33.200 --> 35:35.580]  All right, um, I...
[35:35.580 --> 35:36.800]  I'll get this at the bottom, maybe.
[35:37.080 --> 35:40.800]  I see, uh, ten unique leases on day eight.
[35:41.440 --> 35:47.940]  So, it looks like we had about a hundred, a total of, like, a hundred, eight hundred and forty unique users, um, requesting DHCP leases.
[35:48.280 --> 35:54.120]  Of those 840, uh, some odd, uh, users, we had about 690 of them were from our wireless...
[35:54.120 --> 35:54.700]  Oh, yeah.
[35:54.820 --> 35:55.640]  I didn't explain that to you.
[35:55.680 --> 35:56.640]  So how the hell are you supposed to read it?
[35:56.640 --> 35:56.940]  I'm sorry.
[35:57.180 --> 35:57.500]  Exactly.
[35:57.500 --> 35:57.720]  Thank you.
[35:57.760 --> 35:58.200]  I apologize.
[35:58.600 --> 36:03.440]  So, of those 848 users, uh, 690 or so were, um, wireless.
[36:04.120 --> 36:09.660]  Uh, a hundred, uh, a hundred or so were, uh, wired on the, uh, bring your own computer area.
[36:09.880 --> 36:12.960]  And that third number is, uh, how many, uh, knock people.
[36:12.960 --> 36:14.680]  So how many knock people did we have at the bottom?
[36:15.460 --> 36:16.260]  Thirty-seven.
[36:16.420 --> 36:16.580]  Thirty?
[36:16.700 --> 36:17.400]  On which day?
[36:17.560 --> 36:18.640]  So about 37 knock people.
[36:18.720 --> 36:19.860]  That sounds about, uh, right.
[36:20.440 --> 36:29.900]  So, um, so that shows you that, you know, wireless was a, you know, um, a, um, very important, uh, very substantial part of our network, uh, this year.
[36:30.140 --> 36:35.560]  Um, as opposed to, you know, the past two years where it was either a forethought or we didn't have wireless at all.
[36:36.020 --> 36:39.260]  Um, it shows you how important it was to get this wireless infrastructure working.
[36:39.260 --> 36:45.500]  So I'm glad, uh, Dragron did all that work to get, uh, you know, uh, those 694 wireless users, uh, up and running on the network.
[36:46.120 --> 36:47.340]  Um, yeah.
[36:49.000 --> 36:49.360]  Thanks.
[36:54.700 --> 37:02.980]  One last interesting stat is that, uh, uh, I think 840 is, uh, approximately 40% of our, um, attendance numbers.
[37:03.200 --> 37:05.760]  So, you know, it shows you how important, uh, this net...
[37:05.760 --> 37:08.540]  A lot of people ask, why the hell do you run a network for this conference?
[37:08.580 --> 37:13.020]  Well, you know, there's a good number of people that actually came here and wanted to be on the network, obviously.
[37:13.400 --> 37:13.520]  So...
[37:14.460 --> 37:15.500]  More like 50%.
[37:15.500 --> 37:15.840]  Yeah.
[37:15.960 --> 37:16.460]  How many people...
[37:16.460 --> 37:16.680]  How many...
[37:16.680 --> 37:17.920]  Does anybody know how many people we had?
[37:19.000 --> 37:19.400]  Okay.
[37:19.600 --> 37:20.440]  So it's, yeah, it's closer.
[37:20.660 --> 37:21.760]  It's a little under 50%.
[37:21.760 --> 37:22.880]  So between 40 and 50%.
[37:22.880 --> 37:24.260]  So that's a pretty substantial number.
[37:24.500 --> 37:27.000]  Uh, I think that's about it for the statistics.
[37:27.240 --> 37:27.560]  Uh...
[37:27.560 --> 37:28.500]  There's that graph again.
[37:28.900 --> 37:29.240]  Yeah.
[37:29.320 --> 37:30.500]  We have, uh, three minutes.
[37:30.860 --> 37:33.220]  Um, I, I promised Liza we'd be off here in three minutes.
[37:33.300 --> 37:35.080]  So let's just quickly go through what went wrong.
[37:35.580 --> 37:36.840]  Uh, number one, Verizon.
[37:37.220 --> 37:38.940]  Uh, they really screwed us over this time.
[37:38.940 --> 37:41.020]  Uh, a lot of us are very aggravated with them.
[37:41.600 --> 37:44.020]  Um, they were supposed to bring in the T1s.
[37:44.180 --> 37:49.420]  And apparently, um, well, there, there was, the T1s were supposed to be installed last week.
[37:49.660 --> 37:51.060]  They, uh, said that they did the work.
[37:51.180 --> 37:54.240]  Apparently, uh, they forgot to flip a switch on the smart card.
[37:54.380 --> 38:05.260]  Uh, which, uh, it took, uh, goth chick about, um, 11 hours on the phone, constantly yelling at Verizon dispatch, Verizon customer services, MCI WorldCom, uh, dispatch.
[38:05.260 --> 38:08.720]  Uh, each time, social engineering, more and more information about it.
[38:08.820 --> 38:10.980]  Like, first, all we had was the circuit ID, circuit IDs.
[38:11.440 --> 38:14.720]  We managed to get more contact numbers, more, uh, the trouble tickets on each side.
[38:14.840 --> 38:19.380]  And we got them actually talking to each other, which is, you know, a very rare occurrence, apparently.
[38:19.900 --> 38:22.020]  And, uh, finally, she got a hold of a tech.
[38:22.280 --> 38:23.860]  And the tech told her what the problem was.
[38:23.920 --> 38:25.160]  He was down here in half an hour.
[38:25.480 --> 38:27.920]  We flipped the switch and left, and we had Internet access.
[38:27.920 --> 38:30.000]  And everybody rejoiced.
[38:30.260 --> 38:36.880]  Um, then, unfortunately, some, um, some asshole actually, uh, unplugged our T1 line.
[38:37.120 --> 38:38.760]  And that asshole worked for the building.
[38:39.000 --> 38:40.500]  So, uh, which is very surprising.
[38:41.040 --> 38:43.880]  Um, he was trying to show us where the, uh, T1s were.
[38:44.000 --> 38:45.300]  And he said, oh, here it is.
[38:45.400 --> 38:45.580]  Oops.
[38:45.700 --> 38:46.900]  And he knocked out one of the leads.
[38:47.300 --> 38:49.460]  So, uh, we didn't have Internet access.
[38:49.600 --> 38:52.340]  We had to have, uh, Verizon come down and fix it again.
[38:52.480 --> 38:54.700]  Uh, and get the second T1 up and running.
[38:54.700 --> 38:56.760]  So, uh, finally, they did that.
[38:56.860 --> 38:58.120]  That was about 9 p.m.
[38:58.180 --> 39:01.380]  I guess during the radio show was when the Internet came back up and running.
[39:01.600 --> 39:07.720]  So, uh, I think Verizon forgot to, uh, lock their Verizon closet, uh, in the hotel.
[39:08.000 --> 39:12.140]  And somebody went in and power, hit the power switch on the smart cards.
[39:12.380 --> 39:14.060]  So, everything that T1s came down again.
[39:14.260 --> 39:19.680]  I had, uh, Luis trace the fiber he found, uh, which ran past this closet.
[39:19.860 --> 39:23.400]  He went in and, um, power cycled the, um, the smart cards.
[39:23.400 --> 39:24.320]  We had Internet again.
[39:24.520 --> 39:26.800]  I think that was the last of our Verizon troubles.
[39:27.180 --> 39:34.420]  So, uh, again, this is something next time around we will not discount, we will not, uh, underestimate Verizon's veracity for screwing us over.
[39:35.020 --> 39:40.260]  Um, so I guess the next thing we, uh, we, as you saw, we had some, uh, bandwidth graphs.
[39:40.460 --> 39:44.840]  Uh, I think we could have had nicer bandwidth graphs if we had some better bandwidth management.
[39:44.840 --> 39:50.420]  I forgot, I, with all the chaos, I never got a chance to set up any rate limiting on the, uh, Cisco.
[39:50.620 --> 39:55.060]  So, apparently going to a hacker conference is a good time to, uh, get on PTP networks.
[39:55.360 --> 39:58.620]  So, uh, you know, that really contributed to some of the network slowness.
[39:59.160 --> 40:05.500]  Next time around we intend to rate limit some of that stuff, uh, just so that people could have, you know, more useful, uh, reasons for the Internet.
[40:05.700 --> 40:08.140]  Uh, but, you know, that's something for next time.
[40:08.400 --> 40:12.840]  Um, I think, um, Mike already spoke, Dragon already spoke about the wireless attacks.
[40:12.840 --> 40:14.240]  I don't think there's anything more you want to talk about.
[40:14.240 --> 40:16.040]  Yeah, we saw a couple of, uh, new ones.
[40:16.440 --> 40:22.180]  Um, I logged something in the vicinity of 900 different attacks against the Wi-Fi network.
[40:23.220 --> 40:27.420]  So, of course, a lot of those were one person more than likely, but we got a lot of them.
[40:27.980 --> 40:28.340]  So.
[40:28.680 --> 40:29.860]  All right, we have one more minute.
[40:30.180 --> 40:34.340]  Um, I, the only other thing, uh, I actually am sort of blindsiding Seth here.
[40:34.580 --> 40:37.800]  Uh, the, uh, NOC layout, uh, was pretty good.
[40:37.960 --> 40:42.820]  The one sort of problem we had was that, uh, we had a sort of like a hangout area in the NOC core area.
[40:42.840 --> 40:46.380]  Next time, if we just push that back to where that area, that other area was.
[40:46.520 --> 40:47.000]  On the front of the table?
[40:47.180 --> 40:47.320]  Yeah.
[40:47.500 --> 40:47.620]  Yeah.
[40:47.900 --> 41:00.360]  So, if we just push that, uh, separate a NOC staff area and then a NOC core area where people who actually are working on bringing up the actual network or calling Verizon and yelling at customer service, they can do so peacefully.
[41:00.360 --> 41:09.080]  And then outside of the NOC area where people who are working the NOC staff need to be for supporting the NOC, like, you know, people coming over with questions, they have the space to do that.
[41:09.080 --> 41:19.200]  And also, when they're bored and there's nobody around, they can actually have, you know, hang out and play games and stuff and listen to music, which, uh, we had to become sort of jerks and kicking people out of the NOC staff, unfortunately.
[41:19.400 --> 41:20.820]  But it was the only way we can get stuff done.
[41:20.960 --> 41:22.500]  But next time, it wasn't a major issue.
[41:22.600 --> 41:24.200]  It was just something we were going to change next time.
[41:24.700 --> 41:26.140]  And, uh, 30 seconds.
[41:26.360 --> 41:28.000]  Do you have anything about the PTC that went wrong?
[41:28.000 --> 41:28.500]  Yeah.
[41:28.680 --> 41:31.060]  All that free equipment that's down there, we're giving it all away.
[41:31.200 --> 41:33.000]  Help us out after the closing ceremony.
[41:33.140 --> 41:33.740]  Stop down there.
[41:33.860 --> 41:35.220]  Get a dozen 486s.
[41:35.300 --> 41:38.660]  Almost everything you see on the floor is going to be free and you can have it.
[41:38.880 --> 41:40.580]  Not everything, but most of it.
[41:41.660 --> 41:42.020]  So...
[41:42.020 --> 41:43.040]  I heard 486s.
[41:43.100 --> 41:43.740]  I heard monitors.
[41:43.880 --> 41:44.520]  I didn't hear switches.
[41:44.680 --> 41:45.900]  Did you say anything about switches?
[41:46.140 --> 41:49.240]  Not switches, but probably hubs, um, a lot of other things.
[41:50.000 --> 41:51.760]  Terminals, cabling, token ring stuff.
[41:51.980 --> 41:52.540]  All that's gone.
[41:53.140 --> 41:53.660]  All right.
[41:53.820 --> 41:54.940]  Well, I think that's it.
[41:54.960 --> 41:59.580]  And, uh, just a thanks to a few of the other people who, uh, helped out.
[41:59.780 --> 42:01.760]  Uh, all the NOC staff and volunteers.
[42:02.020 --> 42:02.860]  The Mesh AX guys.
[42:03.300 --> 42:03.960]  Uh, Steve.
[42:04.220 --> 42:04.800]  Raise your hand.
[42:05.000 --> 42:05.740]  Thanks to Steve.
[42:05.960 --> 42:09.000]  He was the ladder monkey who helped get those APs way up there on the curtains.
[42:09.900 --> 42:10.540]  And, uh,
[42:14.280 --> 42:16.160]  everybody who, uh, who loaned us equipment.
[42:16.480 --> 42:18.060]  And everybody who didn't screw the network.
[42:18.320 --> 42:20.560]  And NAC.net for helping us get IP space.
[42:20.860 --> 42:23.420]  And I want to just, uh, yes.
[42:23.420 --> 42:26.240]  Um, I guess we should thank security as well.
[42:26.480 --> 42:27.460]  I forgot to put it out there.
[42:27.800 --> 42:28.700]  But, uh, Frank and security.
[42:28.980 --> 42:33.940]  And actually most of the security team, you know, did a hell of a job helping us, uh, keep the NOC area support, uh, well secured.
[42:39.560 --> 42:46.700]  And one last thing is I, uh, one thing, uh, we have, during the planning, we lost our, uh, the people who were gonna do our wiring.
[42:46.700 --> 42:58.540]  And I just wanted to thank, uh, Luis and, uh, Cy, Amber, uh, for, uh, Mangala and Hitman for really do, everything that, anything that was, uh, electronic was connected to each other.
[42:58.700 --> 43:00.700]  Through a Cat5 cable, that was that team.
[43:00.940 --> 43:03.020]  Uh, they, you know, really did a really good job.
[43:03.020 --> 43:06.100]  I think we had maybe, uh, one, two bad connector connections.
[43:06.480 --> 43:08.000]  I think Hitman did one and I did the other one.
[43:08.180 --> 43:10.520]  So, with all those, uh, Cat5s, we did a pretty good job.
[43:10.840 --> 43:12.320]  So, uh, I think that's it.
[43:12.460 --> 43:13.300]  Lazlow's kicking us off.
[43:13.440 --> 43:14.080]  I wanted to thank everybody.
[43:14.240 --> 43:17.300]  I guess we're not gonna have a Q&A, but you know how to get in touch with us.
[43:17.380 --> 43:19.320]  If you have any other questions, uh, just take us...
[43:19.320 --> 43:19.940]  What's that?
[43:20.600 --> 43:23.700]  Uh, I guess we can put something on the 26...
[43:23.700 --> 43:25.200]  Uh, let me just give you my email address.
[43:25.200 --> 43:26.200]  I guess I'm...
[43:28.620 --> 43:29.500]  Yeah, I'm...
[43:30.160 --> 43:32.200]  Uh, how about spam at...
[43:33.060 --> 43:39.140]  Spam, S-P-A-M, at P-R-A dot T-O.
[43:41.040 --> 43:42.700]  That will work soon.
[43:43.500 --> 43:43.860]  Yeah.
[43:46.640 --> 43:49.300]  Yes, I, you just email that and I'll, I'll put that up.
[43:49.400 --> 43:50.740]  It's on my personal web page.
[43:50.900 --> 43:53.180]  Uh, if you're smart, you can figure out my personal web pages.
[43:53.180 --> 43:54.720]  And it's under my pictures directory.
[43:54.880 --> 43:55.560]  You'll find it.
[43:56.280 --> 43:56.800]  Or...
[43:56.800 --> 43:57.300]  Exactly.
[43:57.640 --> 43:57.880]  All right.
[43:58.220 --> 43:58.860]  That's about it.
[43:59.160 --> 44:00.180]  Uh, thanks, everybody.
[44:00.640 --> 44:01.960]  Thanks for the network, guys.