[00:00.000 --> 00:03.680]  You know enough to be involved and be in the right places.
[01:08.740 --> 01:09.740]  Hello, everyone.
[01:10.280 --> 01:18.280]  I'm here today to share with you a little story, a case history about a company I was engaged with.
[01:19.060 --> 01:26.320]  That, in my opinion, is very important to tell because it describes something that is happening every day.
[01:26.320 --> 01:40.280]  We are used about reading about nations, cyber warfare, huge distributed denial of services against huge companies, multinationals and so on.
[01:40.460 --> 01:50.640]  But in this story, the victim is a real small company, a company like dozens of others exists.
[01:50.640 --> 02:01.640]  And the attack was quite strange, very targeted and, in my opinion, is a cool story that needs to be told.
[02:03.640 --> 02:08.840]  It's a story that, in a good way, we were lucky about that.
[02:08.840 --> 02:15.960]  I will talk about this because I work for a company that usually takes care about security of companies.
[02:16.660 --> 02:26.040]  But security, in my opinion, is not only something that permits me to have money at the end of the month.
[02:26.040 --> 02:50.860]  But it's something that I believe so much that I use all my free time inside a lot of non-profit organizations, whose common time is to share knowledge about this new technology that are supposed to help us and have our life more simpler, but often hide some strange things that could be not so safe.
[02:51.140 --> 02:59.120]  So, discussing it and choosing, knowing what we are choosing, can make these technologies useful.
[03:00.140 --> 03:08.340]  So, the background is that everything starts with a phone call from a number that I didn't know.
[03:08.340 --> 03:16.980]  And on the other side was an unknown guy that was telling me, we have a problem, please help us because we don't know what to do.
[03:17.600 --> 03:33.440]  The guy on the other side of the telephone was the network manager of this company that is a little company that has his business all web-based.
[03:33.440 --> 03:42.820]  It's something like a dashboard where you can exchange messages, for example, I'm selling this, I'm renting this, I'm searching for that, and so on.
[03:43.760 --> 03:49.040]  As you can see, it's a normal company, completely private, held.
[03:50.380 --> 03:54.460]  That is a normal company in my country.
[03:54.740 --> 04:02.960]  So, what is going to happen is happening and is going to happen and happened to a lot of companies like this.
[04:02.960 --> 04:12.640]  The problem was that someone was flooding their web servers and making them unavailable for customers.
[04:13.220 --> 04:18.620]  And this was their infrastructure in the moment in which they were calling me.
[04:18.820 --> 04:22.480]  As you can see, it's very common infrastructure.
[04:23.220 --> 04:32.200]  There is no bleeding edge technology, nor things that are so old and bad.
[04:32.460 --> 04:33.400]  It's normal.
[04:33.660 --> 04:41.900]  And the problem was that the firewall was a quite good firewall, but was not able to handle all the traffic that was arriving.
[04:44.760 --> 04:54.800]  The problem, as we will see, will be also that was not able not to manage the bandwidth, but that packet rate that is quite different.
[04:55.680 --> 05:02.100]  I have also some graphs about how the traffic went during the attack.
[05:02.100 --> 05:15.120]  I don't have all graphs because, as you can understand, during emergency, our urgency was to make the network work and not to save graphs for a conference.
[05:18.880 --> 05:27.000]  Well, we were able to reach a disclosure agreement with the customer to tell you all these things.
[05:27.260 --> 05:31.180]  And because of this, I'm forced to say two or three things.
[05:32.060 --> 05:37.340]  I want to be clear that this attack was a denial of service.
[05:37.340 --> 05:47.020]  So, the website was not reachable, but no data was still or modified, and all users' data were still safe.
[05:47.220 --> 05:50.420]  No one was able to introduce into their servers.
[05:50.620 --> 06:07.680]  Even because during the attack, we were looking before one of our suspects was that someone were using the denial of service as a cover activity for trying to own the servers.
[06:07.980 --> 06:11.240]  But they were not trying to own the servers, only denial of service.
[06:12.600 --> 06:20.260]  For curious guys, how you can run a denial of service attack, you must create your botnet.
[06:20.460 --> 06:27.840]  You must, in fact, as many hosts as you can and make them all create traffic towards a single target.
[06:27.840 --> 06:33.380]  But the fastest way is to find someone that will take money to do that.
[06:33.720 --> 06:42.520]  The funny thing is that paying for that is cheaper than paying a consultant that is defending your network usually.
[06:43.120 --> 06:52.780]  It makes me really laughing to see how flexibility there is in buying the attack from someone else.
[06:52.780 --> 07:03.760]  You know, you can rent a botnet to run the attack by your own, or you can rent the attack directly.
[07:04.060 --> 07:06.020]  Obviously, it's quite more expensive.
[07:06.560 --> 07:14.520]  As you know, this kind of attack is very common and is not focused on any nation.
[07:14.520 --> 07:22.800]  And the number of hosts involved in each attack of this kind is really huge.
[07:23.380 --> 07:30.140]  It's no more like when we were used about one huge server sending a lot of traffic to the target.
[07:30.400 --> 07:33.840]  And so you can ask your Internet provider to filter that out.
[07:33.840 --> 07:46.020]  Now there are a lot of home computers that has not so much bandwidth as common home connections, but all together can do a lot of damages.
[07:46.280 --> 07:57.540]  These graphs represent one of the recent attack of DNS, root DNS servers, and all autonomous systems are colored in a different way.
[07:58.000 --> 07:58.920]  It does quite look good.
[08:00.420 --> 08:11.360]  And one other problem I had during this attack was that many newspapers in Italy were talking about this attack.
[08:12.040 --> 08:16.780]  And as usual, media tend to use the wrong word.
[08:16.780 --> 08:32.100]  So I had also to fight a war in communication to make them understand that it was not so complicated and technological rocket science attack, but was something like any vandals can do, you know.
[08:34.020 --> 08:42.820]  Even one of the more dumb questions at the end of the talk is, you know who were running the attack?
[08:42.820 --> 08:45.640]  The answer is no, for two reasons.
[08:46.880 --> 08:51.560]  The customer had no more money to pay to investigate about that.
[08:52.940 --> 09:08.900]  Policemen were involved to search something about that, but no results at all were found, you know, about this scheme of managing botnet that make really hard to come to the real attack manager.
[09:11.740 --> 09:24.280]  And obviously, a lot of people asked me things like, okay, if you had no police collaboration to analyze any of the infected hosts, could you own one of the hosts?
[09:24.580 --> 09:27.820]  Obviously, yes, but it was not legal in Italy.
[09:28.000 --> 09:29.000]  Yeah, is the correct answer?
[09:29.200 --> 09:29.260]  Yeah.
[09:31.200 --> 09:57.980]  Also, one of the biggest problems when dealing with this kind of attack is that if the network is not designed from the beginning to deal with attacks, with this kind of attacks, you had to follow emergency in every moment and do whatever you can do with what you have in that moment.
[09:57.980 --> 10:16.100]  For example, having an appliance don't have often enough memory to manage some basic blacklists for the huge number of hosts involving the attack, or are not enough flexible to be configured to follow the attack.
[10:17.660 --> 10:20.740]  So, coming to the real context.
[10:21.300 --> 10:28.460]  As I said before, the attack turned out that was a sinful attack.
[10:28.720 --> 10:29.520]  It made me laugh.
[10:29.760 --> 10:32.760]  I was reading about this 20 years ago.
[10:33.300 --> 10:38.040]  It's a really common attack, but in that moment was quite a huge problem.
[10:38.040 --> 10:48.360]  So, our task was to filter out all SIM packets that was not giving the returning acknowledge.
[10:49.180 --> 10:52.080]  And there are a lot of ways to do that.
[10:53.360 --> 11:01.740]  The fate once that I was giving a lecture in a university about OpenBSD in that moment when I received the call.
[11:01.740 --> 11:10.920]  And that day, there with other OpenBSD users talking about how you don't know which strange kind of phone calls I just received.
[11:11.200 --> 11:26.480]  We decided to use OpenBSD to filter all this connection because OpenBSD has in its firewall, as many other technologies, the instruction to filter out SIM fluid and make something like a SIM proxy.
[11:27.200 --> 11:35.540]  So, from that moment, just talking, this was the emergency response team that turned out.
[11:35.820 --> 11:38.920]  So, I was in charge about making decisions.
[11:39.380 --> 11:43.200]  I had a colleague working from another city.
[11:44.980 --> 11:49.260]  Roberto, instead, is this network manager of the customer.
[11:49.560 --> 11:51.240]  And he was in another city.
[11:51.240 --> 11:55.640]  And there was always someone in the data center in another city.
[11:55.840 --> 11:57.940]  So, everything was managed by the phone.
[11:58.140 --> 12:04.000]  I saw in face the customer for the first time two months after the attack was finished.
[12:05.220 --> 12:08.500]  So, deciding the PIX was not okay.
[12:08.820 --> 12:11.020]  We decided to implement OpenBSD.
[12:11.700 --> 12:17.320]  Even here, it was quite funny because they never heard about OpenBSD.
[12:17.320 --> 12:20.260]  They know something about Linux.
[12:21.140 --> 12:24.020]  And we started installing it by phone.
[12:25.080 --> 12:32.940]  And the first installation was quite long because, imagine, press this and now read me on the screen.
[12:33.200 --> 12:33.200]  Yeah.
[12:33.560 --> 12:34.300]  Third option.
[12:34.500 --> 12:34.980]  Press yes.
[12:35.200 --> 12:35.380]  Okay.
[12:35.600 --> 12:36.300]  And go on.
[12:36.440 --> 12:43.000]  Imagine this during a conference like this while you are outside the room waiting for your talk.
[12:43.000 --> 12:51.380]  And then, after the talk, where you are on your car, obviously with the headset, and talking with them, and so on.
[12:52.780 --> 12:57.940]  Obviously, the new firewall was able, in first instance, to filter out everything.
[12:58.260 --> 13:00.160]  Imagine how much I was happy.
[13:00.420 --> 13:01.940]  I received a phone call.
[13:02.120 --> 13:06.200]  And by phone, I was able to solve the problem in some hours.
[13:06.200 --> 13:13.420]  The time needed to make the customer install OpenBSD on a new rack servers that they had in the data center.
[13:14.140 --> 13:15.960]  And everything was working.
[13:16.220 --> 13:18.860]  So, really happy winner, I was thinking.
[13:19.140 --> 13:26.480]  This is the syntax to activate the SimProxy feature on OpenBSD PF.
[13:28.460 --> 13:34.060]  Obviously, this was not the only option to manage an attack like this.
[13:34.060 --> 13:39.000]  There are plenty of products and different approaches that can be used.
[13:39.620 --> 13:45.520]  The first problem was about the Internet provider.
[13:45.780 --> 13:49.180]  It's one of the biggest Internet providers in Italy.
[13:49.520 --> 13:55.800]  But when we asked them to use some approach route-based, they told us, Oh, no, no.
[13:55.900 --> 13:56.400]  Too complex.
[13:56.600 --> 13:59.640]  We don't do things like that.
[14:00.000 --> 14:06.460]  And you will see later on other slides, their support was zero.
[14:07.240 --> 14:08.900]  We were really in difficulty.
[14:09.240 --> 14:19.360]  Imagine when you call the manager in charge in the middle of the night, asking to, for example, reboot one of your servers.
[14:19.620 --> 14:22.640]  And they told you, Okay, I will do that in half an hour.
[14:22.800 --> 14:23.340]  Half an hour?
[14:23.560 --> 14:24.400]  That's why it's so long.
[14:24.680 --> 14:26.160]  I'm how to eat a pizza.
[14:26.440 --> 14:26.860]  Oh, okay.
[14:28.200 --> 14:28.940]  Very bad.
[14:32.900 --> 14:37.920]  So, using our own servers to filter out SimFlow was okay.
[14:38.100 --> 14:38.860]  It was a good approach.
[14:39.720 --> 14:41.200]  We were so comfortable.
[14:42.120 --> 14:44.320]  Everyone was driving back to his location.
[14:44.320 --> 14:47.300]  So, Fabio drove back to Venice, where he lived.
[14:47.520 --> 14:49.340]  And I was going to another conference.
[14:51.600 --> 14:57.500]  It happened that the attacker didn't appreciate our work to filter out his attack.
[15:00.180 --> 15:06.480]  So, he started using more and more bandwidth to generate more and more SimFlow.
[15:06.480 --> 15:09.340]  And that was okay.
[15:09.680 --> 15:13.240]  The server was responding without any problem.
[15:16.000 --> 15:21.240]  Until we were obviously forced to buy more bandwidth from the data center.
[15:21.440 --> 15:27.500]  We were so lucky to be in a data center because having more bandwidth was only about money.
[15:27.940 --> 15:31.080]  We send money, sign something, and more bandwidth.
[15:31.080 --> 15:38.300]  If all the servers were on customer site, there were no way to... or better.
[15:38.860 --> 15:41.560]  Everything would be very longer and more expensive.
[15:42.280 --> 15:45.180]  So, we saw the traffic growing, growing, growing.
[15:45.520 --> 15:49.880]  But feel comfortable because the servers were responding very good.
[15:50.260 --> 15:53.760]  Until there were too much year queue.
[15:53.760 --> 16:06.780]  And to imagine this, if you went to the firewall console and start pressing keys on the console keyboard, no letters were appearing on the monitor.
[16:07.060 --> 16:11.520]  And this was quite difficult to manage.
[16:14.160 --> 16:21.980]  So, for the first thing, we asked the provider to do the only thing they were able to do for us.
[16:21.980 --> 16:24.720]  To filter out international traffic.
[16:26.500 --> 16:29.100]  The company business is Italian based.
[16:29.520 --> 16:33.140]  So, cutting out international traffic seems okay.
[16:33.520 --> 16:35.900]  Because no customers outside Italy.
[16:36.140 --> 16:41.920]  Until you live in Milan and you buy your connection from a company called Swisscom.
[16:42.140 --> 16:43.380]  That is Swiss based.
[16:43.660 --> 16:46.600]  And so, you are international traffic in that moment.
[16:46.920 --> 16:49.420]  So, some customers were cut out.
[16:49.420 --> 16:55.960]  But they are so few that was acceptable while we tried to arrange something.
[16:56.400 --> 16:58.560]  This was not the final solution.
[16:58.680 --> 16:59.120]  Cannot be.
[17:00.640 --> 17:06.540]  So, we decided to split the problem to solve it.
[17:07.200 --> 17:13.240]  We were lucky because they had quite a bunch of new rack servers in the data center.
[17:13.240 --> 17:18.800]  Already Linux installed to be the new web front end.
[17:19.540 --> 17:20.660]  And we told them, okay.
[17:21.120 --> 17:22.240]  Take some of that servers.
[17:23.100 --> 17:24.900]  Put one in the rack.
[17:25.000 --> 17:25.580]  Another one.
[17:26.100 --> 17:27.620]  And install OpenBSD on that.
[17:27.940 --> 17:35.720]  The incredible thing at this point was that the network manager was his second installation of OpenBSD.
[17:35.860 --> 17:36.980]  Was completely autonomous.
[17:37.480 --> 17:38.900]  He installed by himself.
[17:39.200 --> 17:39.740]  Very good.
[17:39.740 --> 17:52.380]  The other problem was that OpenBSD, as every firewall, provides a mechanism to synchronize information such as states, for example.
[17:52.960 --> 18:00.440]  But we didn't want to use any of these mechanisms because having a virtual IP means firewall exchanging traffic.
[18:01.200 --> 18:04.260]  Exchanging states means create more traffic.
[18:04.260 --> 18:07.300]  And our problem was we have too much traffic.
[18:07.620 --> 18:12.220]  So, we installed two different independent firewalls.
[18:13.980 --> 18:24.160]  Again, maybe installing a dedicated Ethernet with CPU on it could solve the problem about managing all that ear cues.
[18:24.540 --> 18:27.560]  But finding it on Sunday was quite difficult.
[18:27.560 --> 18:39.200]  Think that all other phases of the attacks usually happens during night or require hardware that is not normally available in any store street.
[18:41.100 --> 18:46.020]  So, we had to do what was available in that moment.
[18:49.420 --> 19:00.060]  Obviously, also, managing more hosts was not a problem because we started with modifying all files by hands.
[19:00.220 --> 19:06.240]  Then we created some scripts that modifying one files on one host was replicating on every host.
[19:07.980 --> 19:13.020]  So, we continued with PF and with the hardware that was available.
[19:14.660 --> 19:20.140]  The first problem about having two hosts was to send traffic to these two hosts.
[19:20.780 --> 19:21.940]  And it's quite simple.
[19:22.180 --> 19:28.420]  You asked the provider, please, route the traffic with a round-robin algorithm to both our IP.
[19:29.020 --> 19:30.180]  Sure, it's not possible.
[19:30.360 --> 19:32.560]  Oh, your Internet provider is not possible?
[19:32.740 --> 19:36.100]  No, no, we don't create such strange configurations.
[19:38.000 --> 19:39.640]  It was quite a problem.
[19:39.640 --> 19:49.400]  So, we started looking inside the traffic and realized that we were lucky again because the traffic was coming DNS-based.
[19:49.820 --> 19:51.800]  So, they were resolving the name.
[19:51.940 --> 20:02.360]  And it was good because that allowed us to configure inside the DNS two different IP for the website and balancing and round-robin the traffic on two hosts.
[20:03.640 --> 20:15.320]  The attacker never realized that or never take care about we doing that and never switched to attacking directly on IP or we had to change strategy.
[20:15.860 --> 20:18.500]  So, this was working until the end.
[20:19.800 --> 20:24.500]  The other problem to face was about the states because if you send traffic...
[20:25.280 --> 20:29.400]  Okay, I hope I need to see the animation while talking on it.
[20:31.500 --> 20:33.120]  I feel a rock singer.
[20:33.520 --> 20:33.580]  Okay.
[20:36.080 --> 20:41.920]  All traffic coming from users reach, obviously, the customer infrastructure.
[20:41.920 --> 20:47.340]  The problem is that traffic that come back... and here decide what is the default gateway.
[20:47.760 --> 20:53.780]  That come back from the same path is authorized because it exists on the firewall of the state.
[20:54.400 --> 21:01.700]  But the traffic that is asymmetrically routed is blocked because no states exist for that traffic.
[21:02.080 --> 21:04.220]  And this was quite a problem.
[21:04.220 --> 21:05.700]  So, how to solve this?
[21:06.420 --> 21:15.920]  And, as usual, when you have a problem and look like there is no simple solution, you can try to use a different technology...
[21:15.920 --> 21:19.500]  Talk for something different and use it to do that.
[21:20.100 --> 21:27.360]  For example, NAT is used often to NAT the traffic that is going out from your network.
[21:27.360 --> 21:32.700]  But we implemented that for the traffic coming in.
[21:32.940 --> 21:40.780]  So, the result is that all traffic coming from users to the infrastructure look like coming from those two hosts.
[21:41.060 --> 21:42.680]  Only those two hosts.
[21:43.120 --> 21:52.040]  For this reason, no default gateway was needed and all traffic was routed back to the correct firewall that was denetting the traffic.
[21:52.040 --> 22:05.460]  The only con was that we were losing a lot of information about statistics of IP addresses accessing the data center.
[22:05.640 --> 22:10.700]  But we need statistics when your network is working and you're facing an attack.
[22:10.920 --> 22:13.160]  So, we decided it's a good solution.
[22:14.700 --> 22:22.020]  While doing all this, we also work on sysctL and all system parameters to find the data center.
[22:22.020 --> 22:25.960]  To find out something to optimize buffers and so on.
[22:26.120 --> 22:32.380]  To have those two firewalls running more smoothly and faster.
[22:32.820 --> 22:40.400]  So, we asked the provider to take off the filters on international traffic.
[22:41.080 --> 22:44.520]  And everything was working fine.
[22:44.520 --> 22:51.940]  And the bandwidth was more managed than before and was correctly managed.
[22:52.440 --> 22:56.680]  And we saw the traffic raising, raising, raising again.
[23:00.320 --> 23:03.940]  And again, both firewalls get unresponsive.
[23:04.580 --> 23:10.840]  So, as I said before, we had a lot of rack servers there in a corner.
[23:12.480 --> 23:13.240]  Replicate.
[23:13.900 --> 23:18.300]  So, imagine all those people putting firewalls inside the rack.
[23:18.780 --> 23:24.140]  And this network manager from Turing, configuring through a remote connection on another connection.
[23:24.420 --> 23:29.160]  Because the first one was full of traffic installing OpenBSD.
[23:29.740 --> 23:32.400]  Asked the provider to provide more bandwidth.
[23:34.160 --> 23:41.320]  And again, with all those traffic, all those firewalls, we were able to manage the traffic.
[23:41.860 --> 23:42.840]  But look at us.
[23:42.940 --> 23:44.580]  Imagine, guess about us.
[23:44.800 --> 23:46.320]  In front of the monitor.
[23:46.540 --> 23:48.620]  Looking at cacti graphs.
[23:49.240 --> 23:53.540]  Going up, up, up, up.
[23:54.740 --> 23:59.020]  Until it arrived at 850 megabytes.
[24:01.340 --> 24:02.500]  And stays there.
[24:02.940 --> 24:03.860]  And stays there.
[24:04.040 --> 24:04.880]  They are joking.
[24:05.140 --> 24:06.500]  Maybe they have finished money.
[24:06.800 --> 24:08.000]  Have finished hosts.
[24:08.360 --> 24:08.980]  Who knows?
[24:09.240 --> 24:09.960]  Still there.
[24:10.120 --> 24:10.460]  Still there.
[24:10.840 --> 24:11.500]  We won.
[24:11.640 --> 24:12.140]  We are ready.
[24:12.520 --> 24:14.200]  We were so cool.
[24:15.180 --> 24:15.540]  Good.
[24:16.620 --> 24:23.060]  Imagine that all this can be told in some minutes.
[24:23.060 --> 24:24.380]  Like I'm doing now.
[24:25.160 --> 24:27.220]  But imagine it all managed by phone.
[24:27.400 --> 24:29.520]  Because the Turing guy was phoning me.
[24:29.740 --> 24:32.360]  I was phoning the Milan guy that was phoning the Venice guy.
[24:32.640 --> 24:34.300]  The Venice guy was calling me back.
[24:34.480 --> 24:36.680]  Then I was calling the Milan guy back again.
[24:37.080 --> 24:39.260]  And so, 24 hours around.
[24:40.420 --> 24:45.840]  Those are some of the graphs that we were able to collect during the attack.
[24:47.080 --> 24:49.840]  Then we were, again, so comfortable.
[24:50.140 --> 24:50.960]  We were smarter.
[24:51.360 --> 24:52.540]  We outfit everything.
[24:52.540 --> 24:53.540]  Whoa.
[24:54.280 --> 24:54.960]  Wonderful.
[24:55.540 --> 24:59.640]  And then, the traffic starts going down, down, down.
[25:00.020 --> 25:00.620]  I finished money.
[25:00.920 --> 25:01.440]  I hope so.
[25:02.280 --> 25:06.420]  Then, another, some kind of traffic start rising.
[25:06.900 --> 25:07.500]  It's no more.
[25:07.500 --> 25:12.880]  ... is normal HTTP connection.
[25:13.260 --> 25:15.020]  So it was basic.
[25:15.340 --> 25:15.840]  I get fluid.
[25:16.100 --> 25:23.800]  Again, not a technological problem because on OpenBSD, we can rate limit connection.
[25:24.240 --> 25:28.520]  It's not a problem until you know how to rate limit.
[25:28.520 --> 25:31.300]  Because the syntax is very simple, as you can see.
[25:31.480 --> 25:37.260]  You find every host that is exceeding the rate limiting and you put it on blacklist.
[25:37.580 --> 25:40.400]  The real problem is having this number.
[25:40.900 --> 25:49.320]  What is the border of rate limit that cut out the attacker but keep all the customers inside?
[25:50.040 --> 25:59.880]  The task here is not to cut out customers, not to rate limit there, because it's very...
[26:02.500 --> 26:03.820]  Okay, you understood.
[26:07.180 --> 26:10.720]  So we were able to find that parameters.
[26:11.300 --> 26:15.320]  Again, the attack was mitigated.
[26:15.320 --> 26:22.500]  And then we saw that another trend of get fluid was arriving, but was no more.
[26:22.700 --> 26:35.460]  Many hosts continuously refreshing the home page, but was many hosts refreshing at a rate lower than the rate limit.
[26:35.680 --> 26:36.700]  Only one page.
[26:36.960 --> 26:40.240]  That was the page that had two characteristics.
[26:40.240 --> 26:47.700]  It was the worst query on the database and produced the longest page.
[26:47.920 --> 26:54.600]  In that moment, their pages were not divided for, for example, 10 or 20 results a page.
[26:54.740 --> 26:57.280]  It was a single page full of results.
[26:57.540 --> 27:00.260]  Imagine people renting flats on Milan.
[27:00.980 --> 27:02.000]  Huge query.
[27:03.280 --> 27:15.000]  And here was quite a problem because at layer three, you cannot handle with this because you cannot rate limit.
[27:15.340 --> 27:21.500]  You have to go at layer seven and look inside the traffic and decide to filter out these in some way.
[27:21.500 --> 27:23.880]  But it's more applicative.
[27:24.060 --> 27:30.260]  So we start talking with the software house that were managing the balancers and keeping statistics and so on.
[27:30.420 --> 27:32.520]  Asking them if they had any idea.
[27:32.720 --> 27:37.660]  And their manager told me, yeah, yeah, we have an idea, but we need to develop it.
[27:37.840 --> 27:40.300]  So try to do something.
[27:40.680 --> 27:43.540]  Manage the situation for one or two days.
[27:44.160 --> 27:45.800]  We will come with something.
[27:45.800 --> 27:50.280]  Then we start thinking about what to do.
[27:50.720 --> 28:02.760]  And the owner of Bakeca turned out to have some friends that were really suggesting him to buy a filtering service from some company outside.
[28:03.720 --> 28:15.540]  Usually, I tend to disagree with people that want to send their traffic to someone else, even for privacy.
[28:16.540 --> 28:23.740]  But it was an approach wanted by the customer, and we followed them, and we helped them.
[28:24.240 --> 28:28.980]  Remember that we were managing a lot of hosts that were constantly changed.
[28:29.180 --> 28:31.020]  Our blacklists were very long.
[28:31.920 --> 28:45.300]  Having those servers being taught to be web servers, so had gigabytes and gigabytes of RAM helped a lot in managing all huge blacklists.
[28:45.300 --> 28:52.940]  So, as I said before, the software hours need more time.
[28:55.880 --> 29:08.860]  And we start dealing with this traffic laundry that promised that you send us all your traffic, we will clean the traffic and send you back only clean traffic.
[29:08.860 --> 29:15.840]  Well, the first one was command-based support only through Skype.
[29:16.440 --> 29:22.120]  No phone numbers, only a fax number to send a contract signed.
[29:25.380 --> 29:34.080]  We modify the DNS, send them the traffic, they phone back on Skype, telling us, please, please, take your traffic back.
[29:34.080 --> 29:36.260]  We don't want to handle nothing for you.
[29:37.080 --> 29:44.220]  So, okay, that was a strange company, but there exist a lot of professional companies in this field, you know.
[29:44.460 --> 29:51.540]  And I get the customer, choose another company that received all the traffic after we turned the DNS.
[29:51.540 --> 29:58.480]  They call us back, say, this was a serious company with email support, phone support, and so on.
[29:58.720 --> 30:01.640]  They phone back saying, oh, you're really under attack.
[30:01.920 --> 30:02.520]  Oh, f*ck.
[30:05.920 --> 30:16.040]  Yeah, and then, well, the filters that we usually provide are not taught to deal with this kind of huge attack.
[30:16.040 --> 30:26.300]  So, you need our plus contract that take involves a lot of rocket science technology that can...
[30:26.580 --> 30:32.080]  And if you send money, okay, okay, send money back to us and no problem.
[30:33.360 --> 30:36.220]  So, it was really bad experience.
[30:36.220 --> 30:48.940]  I don't want to say that does not exist on the market any company able to manage this kind of attack on their data centers, but our experience was really, really bad.
[30:49.140 --> 30:54.800]  Those are the graphs that one of these laundry send us.
[30:55.080 --> 30:57.160]  We are really facing an attack.
[31:00.110 --> 31:18.870]  So, while we were playing with DNS to send them traffic, the software house was able to identify inside the get fluid some parameters to cut out at balancer layers the get fluid on that specific page.
[31:19.270 --> 31:22.770]  They use, obviously, some application technology.
[31:22.770 --> 31:33.610]  Again, nothing so complex or incredible, but the risk is always that when you apply these, you cut out customers.
[31:33.970 --> 31:35.650]  And if you do that, you're losing money.
[31:36.710 --> 31:54.030]  While doing that, the other thing we did was to install a second MySQL server as database to avoid a single point of failure, but also to host load balancer to obtain more performance.
[31:54.530 --> 31:59.950]  And this helped a lot in mitigating the attack.
[32:00.570 --> 32:05.270]  And again, the infrastructure was up and running really smooth.
[32:06.450 --> 32:09.430]  We were really, really tired.
[32:09.790 --> 32:14.110]  At this point, two weeks were passed.
[32:16.170 --> 32:23.030]  And I think that if my wife didn't decide to broke with me during this situation, we'll never do.
[32:23.630 --> 32:30.410]  Because imagine that I was sleeping with a headset in my hair and phone under the pillow.
[32:30.710 --> 32:35.450]  And I went to the restaurant with laptop under... with me.
[32:35.610 --> 32:40.330]  And in the middle of the dinner, phone call, so open the laptop and connect.
[32:40.330 --> 32:41.170]  And so on.
[32:41.450 --> 32:43.550]  Every day, every time, every hour.
[32:45.510 --> 32:49.830]  But finally, the infrastructure was up, running, really smooth.
[32:50.110 --> 32:51.050]  We are the winner.
[32:52.870 --> 32:53.410]  Party.
[32:53.890 --> 32:54.230]  Yeah.
[32:54.490 --> 33:02.670]  Party a lot until in the middle of the night, ring your phone call saying, Oh, I forgot to tell you one thing, told me the network manager.
[33:02.670 --> 33:06.190]  The DNS servers are not in the same data center.
[33:06.350 --> 33:07.750]  They are in another place.
[33:07.890 --> 33:09.910]  And they have a very few bandwidth.
[33:10.390 --> 33:13.250]  So, they are flooding the DNS.
[33:13.530 --> 33:14.230]  So, no problem.
[33:14.470 --> 33:19.570]  We install bind on... or some other DNS, I don't remember.
[33:20.750 --> 33:22.790]  Inside the host in the infrastructure.
[33:23.310 --> 33:30.470]  And had two reliable DNS with one gigabit bandwidth and some firewall in front of them.
[33:30.470 --> 33:32.650]  And everything was running smooth again.
[33:33.170 --> 33:35.290]  And again, we thought...
[33:37.830 --> 33:41.150]  When everything was working good, we thought we can have a party.
[33:41.450 --> 33:43.050]  And they had finished.
[33:43.430 --> 33:48.570]  Well, one of the satisfaction of all this is that I had no sleep at all.
[33:48.570 --> 33:51.190]  But the guy on the other side has no too.
[33:51.510 --> 34:00.130]  Because his reaction time was often about two or three hours after our configuration.
[34:00.630 --> 34:03.090]  So, on the other side, they were not sleeping too.
[34:04.730 --> 34:08.030]  Another little detail about how the site works.
[34:08.710 --> 34:12.110]  For private users, the site is for free.
[34:12.110 --> 34:17.510]  And everyone can go on the site and create a new announce.
[34:19.390 --> 34:21.810]  You only need to insert his email.
[34:22.050 --> 34:26.770]  You receive an email saying you just insert this announce with these details and so on.
[34:26.950 --> 34:30.410]  If you confirm the email, your announce is published.
[34:31.130 --> 34:36.450]  And then the attacker created a script that inserted a lot of announcements.
[34:37.330 --> 34:41.510]  The problem about that was not having all the announce on the queue.
[34:42.170 --> 34:48.110]  Because with some scripting, you can easily find them and cut them out.
[34:48.510 --> 34:52.610]  The problem was that every announce was generating an email.
[34:52.850 --> 34:55.550]  And the email server's queue was full.
[34:55.790 --> 35:04.510]  And there are not enough details inside the email to search in the queue those emails to be deleted.
[35:04.510 --> 35:14.090]  So, the only way was to find a really fast solution to get all that emails to go away.
[35:14.490 --> 35:15.970]  I'll tell you a secret.
[35:16.370 --> 35:17.590]  A very bad thing.
[35:18.030 --> 35:24.570]  It's more than 10 years that I complain with other vendors, with customers and so on.
[35:24.750 --> 35:30.050]  That say, did you look at my new wonderful device?
[35:30.050 --> 35:37.170]  I bought this new firewall that hacked also as a fax server, file server, email server, and so on.
[35:37.330 --> 35:37.430]  Okay?
[35:37.710 --> 35:38.530]  That's not a firewall.
[35:38.730 --> 35:39.070]  It's a shit.
[35:39.290 --> 35:39.710]  But...
[35:42.790 --> 35:44.690]  I think you will agree with me.
[35:44.810 --> 35:46.170]  A firewall is a firewall.
[35:46.310 --> 35:49.070]  Until his only task is to filter packets.
[35:49.270 --> 35:51.630]  He cannot do anything else.
[35:51.630 --> 35:59.010]  But having the hacked servers that were only filtering traffic was too tempting.
[35:59.630 --> 36:08.370]  And we activated the send mail on every host and configured the mail servers to use all eight hosts to relay emails.
[36:08.590 --> 36:14.250]  In this way, we were able to split the queue and have everything going out.
[36:14.250 --> 36:21.790]  But even because those send mails is OpenBSD send mails, so it's really different from the normal one.
[36:22.090 --> 36:23.590]  It's quite more secure.
[36:25.070 --> 36:28.590]  We're configured to be very aggressive in timing.
[36:28.970 --> 36:35.250]  Even because, as you can imagine, most of email had non-existent domains, non-existent users, and so on.
[36:35.370 --> 36:38.890]  So we're generating a lot of bounces, and so on.
[36:39.030 --> 36:41.470]  And this was working very good.
[36:42.770 --> 36:49.730]  After all these emails flow out, we lived for many days thinking, And now?
[36:50.270 --> 36:54.150]  If I would be the attacker, I would do this, I would do that.
[36:54.430 --> 36:57.190]  Another option could be doing that.
[36:57.810 --> 37:01.550]  Lucky ones that maybe they have finished money or something else.
[37:01.930 --> 37:05.790]  Nothing happens in the day after this.
[37:05.790 --> 37:12.050]  So this was the last, in the middle of the night, call I received.
[37:13.390 --> 37:22.290]  About all this situation, the thing that really I found amazing was the approach of the customer with the problem.
[37:23.610 --> 37:26.570]  I said before, they are a marketing company.
[37:26.810 --> 37:31.130]  Maybe they know nothing about IT or too much.
[37:31.410 --> 37:39.710]  Usually, you know, when your company is under attack, the approach is, okay, let's hide everything.
[37:40.130 --> 37:40.850]  Let's say, attack?
[37:41.210 --> 37:42.170]  Which attack?
[37:42.470 --> 37:43.350]  No one is attacking.
[37:43.350 --> 37:46.910]  Oh, it's just a hardware fault or something like that.
[37:47.230 --> 37:51.390]  He decided to approach the problem in a complete different way.
[37:51.630 --> 37:55.090]  He started advertising the attack.
[37:56.510 --> 37:59.310]  Buying advertisement on newspapers.
[38:00.370 --> 38:02.470]  Talking with bloggers.
[38:02.470 --> 38:02.670]  Talking with bloggers.
[38:03.170 --> 38:09.730]  Creating a media campaign saying, our site is free for everyone.
[38:10.590 --> 38:15.150]  Someone don't want our site to be available to everyone.
[38:15.670 --> 38:19.030]  Someone don't want freedom on Internet.
[38:19.650 --> 38:20.730]  Fight with us!
[38:21.250 --> 38:24.590]  It was a real cool approach.
[38:24.590 --> 38:30.330]  They paid a lot of newspapers to interview me as the expert.
[38:30.970 --> 38:32.810]  To describe the attack.
[38:33.170 --> 38:35.930]  To explain what was happening.
[38:36.610 --> 38:38.970]  And even some technical details.
[38:40.490 --> 38:48.390]  Asking for the customers to support them because they were in difficult because of their believing in freedom.
[38:48.790 --> 38:51.810]  This was a really cool move.
[38:51.810 --> 38:57.550]  Because not only they didn't lose any customer.
[38:57.790 --> 39:02.190]  But they get new customers because of all those advertisements.
[39:02.830 --> 39:06.190]  So, in my opinion, this approach was really cool.
[39:06.710 --> 39:10.850]  I think that in the first month.
[39:11.290 --> 39:15.810]  So, during the last two weeks of the attack.
[39:15.810 --> 39:17.750]  And other two weeks.
[39:17.950 --> 39:24.390]  200 people wrote on the site they opened about the attack.
[39:24.770 --> 39:26.150]  We support you.
[39:26.430 --> 39:27.050]  We are with you.
[39:27.190 --> 39:28.250]  Internet is for freedom.
[39:28.510 --> 39:28.870]  And so on.
[39:29.270 --> 39:34.350]  200 people that decided to use their time to support this company.
[39:34.610 --> 39:41.110]  Was a really amazing example of incident management, in my opinion.
[39:43.770 --> 39:45.830]  So, we are near conclusion.
[39:47.910 --> 39:52.710]  You remember that the infrastructure, when we started.
[39:53.870 --> 39:58.110]  When we were at this point, this was the new infrastructure.
[39:59.850 --> 40:03.650]  Now, everything is changed from this point.
[40:03.870 --> 40:15.830]  Because when the emergency finished, we took the time to sit down and design a network that was taught to manage this, to afford this kind of problems.
[40:16.070 --> 40:20.450]  So, they are no more in that data center where people don't know what to do.
[40:20.450 --> 40:24.550]  They are, they have a different infrastructure.
[40:24.930 --> 40:27.650]  So, all these details are now obsolete.
[40:30.930 --> 40:40.670]  And, as you see, a lot of traffic, but mainly a lot of hosts were involved in this.
[40:40.870 --> 40:46.230]  It was quite a big botnet that we saw during this attack.
[40:46.230 --> 40:51.510]  As I said before, we were lucky from different point of view.
[40:53.030 --> 40:56.750]  From a technical point of view, because the attack was DNS based.
[40:56.870 --> 40:58.370]  We were in data center and so on.
[40:58.810 --> 41:06.510]  But also, because we were talking with a company that had enough money to protect their business.
[41:06.510 --> 41:16.470]  If they didn't have enough money to pay one gigabit bandwidth, the attack was successful and no one can do anything else.
[41:17.570 --> 41:20.430]  They were able to pay consultants.
[41:20.810 --> 41:27.330]  Often, exist a lot of companies that prefer to stay down than to pay something to manage the situation.
[41:27.330 --> 41:39.190]  And something that I really appreciated was that all people I was working with, except for the ESP people, were really smart.
[41:39.550 --> 41:45.110]  From the network manager that became an OpenBSD guru in a week.
[41:45.110 --> 41:54.330]  And all people that were like me, responding to the phone every hour, every moment, always, always, always.
[41:57.010 --> 42:03.090]  And obviously, when you have to manage with this kind of attack, it's always a nightmare.
[42:03.090 --> 42:07.450]  When you are leaving that on your skin, it's very painful.
[42:08.530 --> 42:11.870]  Because in many moments...
[42:11.870 --> 42:16.570]  Now, talking about that, I gave you what we did.
[42:16.810 --> 42:20.690]  But when you are there, alone, in the middle of the night, thinking, And now?
[42:20.850 --> 42:21.830]  What can I do now?
[42:22.250 --> 42:24.510]  It's quite painful.
[42:26.470 --> 42:33.070]  But everyone involved in this adventure, at the end, was really smiling and happy.
[42:33.070 --> 42:37.170]  To say, we were able to defeat the evil.
[42:39.870 --> 42:48.570]  Another interesting aspect of this is, as I said before, I went to a lot of conferences during those weeks.
[42:48.930 --> 42:54.450]  And, you know, often, you don't remember everything about syntax by memory.
[42:54.690 --> 43:08.710]  And so, imagine me, by phone, outside here, walking, talking with the customers, and then asking whoever passed, do you remember that option of TCP dump that helped me in finding that particular...
[43:08.710 --> 43:17.730]  So, a lot of people were helping out only with a suggestion or because they were reminding about something.
[43:18.270 --> 43:22.610]  A lot of people, friends, reading on newspapers was happening.
[43:22.610 --> 43:24.210]  They mailed me, phoned me.
[43:24.350 --> 43:25.830]  Do you need any help?
[43:26.250 --> 43:28.130]  I have something that can help you.
[43:28.590 --> 43:30.670]  A friend that is the...
[43:30.670 --> 43:32.710]  I don't know if you know...
[43:32.710 --> 43:35.830]  And TOP is the program to network and monitor, yeah?
[43:36.150 --> 43:38.110]  Luca Derry is a friend of mine.
[43:38.550 --> 43:44.870]  And he phoned me to ask, Oh, I'm testing some 10 gigabytes network card with CPU on it.
[43:45.030 --> 43:46.770]  If you want, I have one here.
[43:46.770 --> 43:47.830]  I can send it.
[43:48.070 --> 43:51.630]  So, a lot of support, even more for a lot of people.
[43:52.270 --> 44:01.370]  And those two guys get a lot of rants because they are colleagues that I was calling during night, telling them, Oh, and now what can we do?
[44:01.770 --> 44:02.670]  And so on.
[44:03.250 --> 44:08.830]  Also, I want to finish with a little photo that I really love.
[44:09.050 --> 44:12.870]  That is a stitch as Emperor Papatine that says, Don't be evil.
[44:15.850 --> 44:20.370]  So, I try to be quite faster to have some more time about question.
[44:22.750 --> 44:23.350]  Please.
[44:23.610 --> 44:25.970]  Did you ever figure out who was attacking or why?
[44:26.110 --> 44:27.350]  I'm recorded, yeah?
[44:32.790 --> 44:33.390]  Well...
[44:33.390 --> 44:35.650]  Look like some...
[44:35.650 --> 44:37.870]  We have no proof at all.
[44:38.170 --> 44:40.130]  It's just my imagination.
[44:40.130 --> 44:51.630]  But some weeks before the attacks start, the owner of the company received an offer from a competitor to be bought.
[44:52.070 --> 45:01.530]  And he, in a real wrong way, told them, Well, wait, we are too small now.
[45:01.870 --> 45:05.770]  When we will be more expensive to buy, then come back again.
[45:07.810 --> 45:15.490]  I don't know if those two things are related, but it was the only option we had in our mind.
[45:17.730 --> 45:28.670]  Did the Dell servers you used as the BSD firewalls have copper server NICs with co-processors on them, or were they just the standard NICs on them?
[45:28.670 --> 45:29.910]  Standard, standard.
[45:30.330 --> 45:40.110]  As I said before, they just bought 10 of them to put in the rack as new web servers, so they were legacy servers.
[45:40.570 --> 45:45.890]  As I said, we did with what was available in that moment.
[45:47.510 --> 45:48.150]  Please?
[45:52.930 --> 46:10.290]  We tried to figure out the only certain data we were able to track was the contemporary hosts generating traffic were 20,000.
[46:11.290 --> 46:15.890]  But they were rotating in something like 15, 20 minutes.
[46:18.270 --> 46:28.650]  If a host that was appearing in the morning was coming back in the afternoon, we were not able to have a metrics to track that.
[46:28.650 --> 46:50.750]  But in some moments, the blacklist were over 200,000 lines, hosts, but we were cleaning that because of tests.
[46:51.570 --> 46:56.830]  Often customers went in, so no affordable, not trusted data.
[46:57.550 --> 46:58.070]  Please?
[46:58.290 --> 47:01.130]  It seems that in this case, the ISPs...
[47:01.130 --> 47:01.730]  Louder.
[47:01.910 --> 47:10.050]  It seems that in this case, the ISPs do not have any interest in helping you out because you would have bought more bandwidth than if they had any of them.
[47:10.050 --> 47:11.210]  Maybe not.
[47:11.770 --> 47:12.350]  I don't know.
[47:12.470 --> 47:17.150]  I don't know, just in my imagination, but most of the ISPs are at least here in the U.S.
[47:17.310 --> 47:21.310]  Because we also are definitely, they are responsibly based.
[47:21.510 --> 47:23.130]  Some people have certain responsibilities.
[47:24.390 --> 47:25.030]  No.
[47:26.450 --> 47:38.550]  From a low point of view, the only information they gave us was if these things go too far, we will cut you out because you're...
[47:38.550 --> 47:41.790]  Maybe you will damage other customers in the data center.
[47:42.070 --> 47:45.190]  It was the only thing they say about the contract.
[47:45.810 --> 47:55.630]  From a technical point of view, their only answer were, or, we don't do that, or, what are you talking about?
[47:57.810 --> 48:04.610]  And it's one of the, it's not the biggest, but one of the top five data centers in Italy.
[48:06.370 --> 48:06.770]  Please?
[48:07.590 --> 48:15.190]  In your opinion, if we took a box, put high-end nicks in it, you know, how much data could we filter with software?
[48:16.050 --> 48:17.570]  You know, if we built a box?
[48:19.670 --> 48:21.130]  Can you repeat slowly, please?
[48:21.770 --> 48:30.030]  So, if we were to take a box, put high-end nicks in it, you know, as many processors as we thought we needed, how much traffic do you think it would handle?
[48:30.030 --> 48:32.550]  We didn't make any tests.
[48:33.430 --> 48:43.150]  Legacy machine we saw at 180 megabytes drop connections, but we didn't have time to put more processor or change something.
[48:43.350 --> 48:43.570]  Okay.
[48:44.770 --> 48:45.690]  Just one moment.
[48:45.830 --> 48:47.090]  There was a guy from before.
[48:51.430 --> 48:53.770]  Nine months ago, I was in this accident situation.
[48:54.270 --> 48:56.110]  I just want to say a small problem.
[48:56.390 --> 48:56.510]  Sorry.
[48:57.530 --> 48:59.650]  I implemented roughly the same solution.
[49:01.570 --> 49:03.390]  It's a great operating system.
[49:03.390 --> 49:07.150]  Anyone that's a hard worker, it's just a little bit of a difference.
[49:08.110 --> 49:08.870]  I did documentation.
[49:09.910 --> 49:12.110]  I also had this same trouble.
[49:12.610 --> 49:14.170]  I'm gonna do the protection.
[49:14.430 --> 49:16.010]  I have the foundation of pension companies.
[49:16.510 --> 49:17.830]  They're really sketchy.
[49:18.130 --> 49:19.410]  They like to buy a new block.
[49:19.790 --> 49:21.190]  Be very effective with them.
[49:21.650 --> 49:27.230]  They can help offer new solutions for companies that account hard for me to actually have less .
[49:29.090 --> 49:33.010]  And as far as DNS, we just run DNS, we'll help people in.
[49:46.440 --> 49:55.640]  Did you offer to help the providers knock with reconcrued providers, and so what was the response?
[49:59.610 --> 50:00.500]  Let's say yes.
[50:03.260 --> 50:08.580]  I already had, in the past, some experience with them.
[50:10.660 --> 50:18.460]  I gave them some consultants for the new data center that they were opening.
[50:19.860 --> 50:27.920]  But they had never opened that data center, just booked the walls and designed the network.
[50:28.280 --> 50:35.080]  So we already know each other, but at the end of the story, nothing goes further.
[50:36.900 --> 50:41.100]  I offered, but, you know, if I help them, I want money back.
[50:41.480 --> 50:43.720]  And they decided it's not okay.
[50:45.420 --> 50:45.860]  So...
[50:46.760 --> 50:50.240]  So currently, it seems like maybe there's not a...
[50:50.240 --> 50:52.320]  We'll wait, because I don't hear.
[50:53.060 --> 50:55.180]  So far, there's maybe no...
[51:08.820 --> 51:17.720]  But in that moment, being in an OpenBSD conference helped me a lot in deciding about OpenBSD.
[51:17.720 --> 51:27.460]  Even commercial, my company, in my company, I decided to use OpenBSD because I find it more flexible from this point of view.
[51:27.700 --> 51:36.620]  Because usually, if you want to install a firewall, for example, a normal distribution, you have to install and then take away things.
[51:36.620 --> 51:40.420]  In OpenBSD, usually, you install and add whatever you need.
[51:40.960 --> 51:42.880]  Because it's more clean.
[51:43.380 --> 51:45.620]  I prefer it, but it's like religion.
[51:46.040 --> 51:46.920]  It's not true.
[51:47.120 --> 51:47.780]  It's my preference.
[51:52.100 --> 51:53.740]  It's my preference.
[51:54.180 --> 51:55.080]  Other question?
[51:58.680 --> 52:01.780]  What was your company's relationship to...
[52:03.060 --> 52:05.780]  Bacheca was a customer of my company.
[52:06.240 --> 52:11.360]  Here, I decided to take away my company's science and so on because I don't want these to be...
[52:11.360 --> 52:13.860]  No, I understand, but did you build their web app?
[52:14.140 --> 52:15.040]  Or were you the security...
[52:15.040 --> 52:15.540]  No, no, no.
[52:16.100 --> 52:16.880]  They had...
[52:18.240 --> 52:22.420]  During the attack, they are in Turin, in Italy.
[52:22.860 --> 52:32.280]  And in Turin, live one of my best friends, that is Raul Chiesa, that was a friend of the owner of this.
[52:32.920 --> 52:38.140]  And he called Raul, asking, Raul, I know that your company don't need security.
[52:38.560 --> 52:39.780]  Can you do something for me?
[52:39.940 --> 52:45.460]  He told them, no, our company do other things, but I have a friend that do this.
[52:45.460 --> 52:47.440]  And gave them my phone number.
[52:47.920 --> 52:50.660]  So, we were engaged with my phone.
[52:50.780 --> 52:53.800]  So, you have security consultants, basically, as far as the crime, so...
[52:53.800 --> 52:54.320]  Yeah, yeah.
[52:54.620 --> 52:56.380]  It was a consultant for them, yeah.
[52:59.500 --> 52:59.980]  Okay.
[53:00.220 --> 53:04.540]  If there are no more questions, I can close, so the next speaker can arrange the podium.
[53:04.540 --> 53:05.600]  Thank you very much.
[53:05.900 --> 53:06.400]  Thank you very much.
[53:14.500 --> 53:19.980]  And WSU, feel free to write me, or follow me on Twitter, or add me on Facebook, and so on.
[53:31.340 --> 53:31.820]  Wow.
[53:32.060 --> 53:32.260]  Walk in...