[00:00.000 --> 00:06.400]  So, our talk is Protecting JetBlue Airways from Cyberthreats in the Cloud, and I'm just going to introduce myself.
[00:06.620 --> 00:07.260]  I'm Randy Lorraine.
[00:07.420 --> 00:11.600]  I'm the manager of security engineering and a security architect at JetBlue.
[00:12.360 --> 00:15.920]  Prior to that, I was a network engineer for seven years at J.Crew.
[00:15.980 --> 00:18.460]  They're a retailer with about 500 stores in the U.S.
[00:19.020 --> 00:21.140]  And prior to that, I wasn't in tech at all.
[00:21.320 --> 00:26.180]  I was a pianist, a teacher, a piano salesman at Kauai and Yamaha.
[00:26.420 --> 00:29.980]  So, you know, when I was in college, I was studying computers, working in the showroom.
[00:29.980 --> 00:31.520]  paying off my debt that way.
[00:31.680 --> 00:34.340]  And then I transitioned over into tech about 10 years ago.
[00:35.040 --> 00:36.760]  And outside of tech, I love to travel.
[00:36.980 --> 00:38.740]  You know, working for an airline, we fly for free.
[00:39.100 --> 00:40.440]  I've been to 73 countries.
[00:40.880 --> 00:41.760]  I love performing.
[00:41.980 --> 00:44.960]  I might even be playing with DJ Spock at 2 o'clock outside.
[00:46.200 --> 00:49.340]  And then, you know, I love hacking, learning how things work.
[00:49.700 --> 00:50.680]  And I'll introduce Greg.
[00:52.120 --> 00:52.720]  Hey, guys.
[00:52.840 --> 00:53.460]  I'm Greg Speranza.
[00:54.240 --> 00:56.040]  I'm security engineer at JetBlue.
[00:56.680 --> 00:58.400]  Pretty much started my career at JetBlue.
[00:58.580 --> 00:59.920]  Been here for seven years.
[00:59.980 --> 01:03.380]  Started out as desktop support.
[01:03.780 --> 01:05.680]  Worked my way up to security analyst.
[01:05.980 --> 01:07.540]  And now, obviously, I'm a security engineer.
[01:08.860 --> 01:13.140]  Kind of like Randy in a past life, I wasn't doing anything with tech at all.
[01:13.340 --> 01:16.280]  I was trying to get into audio engineering and music and all these things.
[01:16.440 --> 01:17.360]  But obviously, I fell through.
[01:17.560 --> 01:18.340]  Got into tech.
[01:19.340 --> 01:21.980]  Outside of work, I'm a big music and film enthusiast.
[01:22.220 --> 01:23.500]  I play multiple instruments.
[01:24.220 --> 01:25.980]  Also a proud dog and cat dad.
[01:27.500 --> 01:27.900]  Cool.
[01:28.500 --> 01:30.540]  So our talk is going to have four sections.
[01:30.860 --> 01:35.100]  The first section, we're going to show, you know, how JetBlue operates from an IT perspective.
[01:35.520 --> 01:36.960]  What are the challenges we see?
[01:37.080 --> 01:38.180]  What makes an airline unique?
[01:38.580 --> 01:44.380]  Greg is going to talk a little bit about how, you know, he does a good job managing the WAF and, you know, protecting our site from bots.
[01:44.580 --> 01:47.180]  The travel industry is heavily botted, as you can imagine.
[01:47.180 --> 02:02.080]  We're also going to talk about the importance of cyber intelligence, where JetBlue is a founding member of the Aviation ISAC, and, you know, this is a forum, a private forum, where all airlines, OEMs, manufacturers can come together and collaborate, share intelligence, and just share, you know, networking with each others.
[02:06.120 --> 02:08.220]  And then at the end, we're going to have some time for Q&A.
[02:08.220 --> 02:08.220]  Okay.
[02:09.680 --> 02:12.780]  So what makes an airline unique from an IT perspective?
[02:13.240 --> 02:18.280]  So I've worked in typical enterprise, you know, you know, banks, retail, you know, those are all good.
[02:18.380 --> 02:19.040]  They have a lot of money.
[02:19.160 --> 02:20.060]  They can do a lot of things.
[02:20.200 --> 02:21.100]  The airline, you can't.
[02:21.240 --> 02:26.340]  You have to secure, you know, a company that doesn't make much profit, and it's running around the clock.
[02:26.540 --> 02:29.720]  When something breaks, things compound very quickly.
[02:29.880 --> 02:31.120]  There's a huge snowball effect.
[02:31.120 --> 02:33.380]  So say, you know, your booking system goes down.
[02:33.500 --> 02:35.040]  Now people can't book flights.
[02:35.040 --> 02:39.240]  Or if you have latency to your check-in systems, people can't check into their flights.
[02:39.240 --> 02:40.200]  The flight gets delayed.
[02:40.300 --> 02:41.600]  Eventually, the crew times out.
[02:41.740 --> 02:46.500]  That compounds over a schedule, over 24 hours, and, you know, we lose a lot of money.
[02:46.620 --> 02:51.500]  So I think it's, you know, several million dollars an hour for some of these major systems when they go down.
[02:51.980 --> 02:55.620]  The industry is also one of the most heavily regulated industries on earth.
[02:55.620 --> 03:03.920]  But, you know, everything you do, you have to check with the FAA, the DHS, SOCs, PCI, so we have so many regulations, both external and internal.
[03:04.160 --> 03:10.520]  When you want to change things, everything has to go through multiple change control boards before they can get pushed out.
[03:10.660 --> 03:15.020]  So that becomes a challenge for security, because, you know, you have to be able to respond quickly to threats.
[03:15.900 --> 03:20.780]  Greg can also give some insight about how, you know, it's hard it is to secure some of the devices that we have.
[03:21.440 --> 03:38.520]  Yeah, back, uh, in my desktop support days, sorry, you're sharing the mic, um, I managed the iPads for our in-flight, in-flight crew members, um, and, you know, with that being said, these are thousands of devices that, you know, um, that team that I was on is managing,
[03:38.980 --> 03:50.460]  um, and obviously we're an airline, so these people are traveling all over the country, um, now, you know, they're traveling, um, into a decent part of Europe, now that we fly to Europe, several different countries.
[03:50.780 --> 04:12.820]  Um, there was a, there was a few major issues and still are, um, one of them being, um, equipment theft and them losing the mobile devices, um, and we do have controls in place, like, um, you know, requiring a certain strong password, like six digits or whatever it is,
[04:12.940 --> 04:19.620]  but, you know, people get lazy, they, they, uh, they just will put six zeros or whatever, you know, things like that, um, of course.
[04:19.620 --> 04:35.540]  Uh, so if they were to lose it or it did get stolen, which actually happens quite often, probably, like, easily a couple times a week, if not more, um, you know, if something, if the device would fall into the wrong hands, um, and they, you know, potentially someone,
[04:35.620 --> 04:42.820]  that person breaks into the iPad by guessing the passcode or whatever, um, then they have access to all this, uh, you know, sensitive data.
[04:42.820 --> 04:53.160]  Um, especially pilots, you know, they have certain applications, mobile applications on the iPads that contain, uh, things like our flight routes, um, and confidential documents.
[04:53.160 --> 04:55.560]  So, you can see how good, how this could be a challenge.
[04:57.480 --> 05:04.160]  Thankfully, our iPads don't have much of them and they're kind of slow, but, um, we also have to do a lot with redundancy.
[05:04.500 --> 05:09.360]  Um, like I've said earlier, you know, if there's any outages, uh, it can affect business in a significant way.
[05:09.780 --> 05:19.740]  So there's, as an architect, when we're integrating or putting new systems in or going to new clouds, we have to have a lot of redundant systems and design them with stability and security in mind.
[05:20.520 --> 05:27.040]  So, you know, those are the kind of the four pillars of operating the business from a cyber perspective, what are the challenges?
[05:27.360 --> 05:30.900]  So, you know, if we get ransomware, you know, we're screwed, nothing can run.
[05:31.040 --> 05:33.000]  So we have to have segmentation in place.
[05:33.280 --> 05:39.980]  There's also a lot of third-party systems that we integrate with for fuel, for satellite connections, for, you know, your website where it's hosted.
[05:40.160 --> 05:42.960]  So if, if they get compromised, we get compromised.
[05:43.140 --> 05:48.600]  So third-party risk is huge in our industry because there's so many, you know, third-party connections we have.
[05:48.600 --> 05:50.360]  It's not a very closed environment.
[05:50.860 --> 05:57.600]  Uh, you know, there's also regulation that kind of slows down our response, um, whether that's external or internal.
[05:57.960 --> 06:04.920]  So, you know, getting, getting around that or working with the board or legal to get, you know, some speed for, for security, uh, definitely helps.
[06:05.460 --> 06:15.900]  And then like Greg mentioned before, equipment theft, uh, we've put a lot of controls in now to, you know, not have a lot of data on the actual iPad itself, since we have so many fluid devices moving around the network.
[06:16.400 --> 06:20.000]  And then we do a fair amount of, um, DR testing to make sure that we have stability.
[06:20.680 --> 06:23.580]  So these are some of the ways we solve some of these problems here.
[06:23.800 --> 06:25.420]  So, you know, we have defense in depth.
[06:25.860 --> 06:29.660]  Uh, you know, when we have a lot of regulation, we usually spin up an incident.
[06:29.980 --> 06:34.880]  Uh, we have an incident leader and they can quickly make changes so that we can, you know, make the paperwork after we do the change.
[06:34.880 --> 06:46.760]  Uh, if we have a major security incident, typically that gets spun up, we get looped in, legal gets looped in, and then things move a lot quicker because you have to react in minutes now and you have to react with automation, uh, cause the attackers are using automation.
[06:47.720 --> 06:50.840]  So I'm going to take you guys into the structure of our team.
[06:51.380 --> 06:53.600]  Um, and they have a lot on this slide.
[06:53.720 --> 06:56.720]  I'm not going to go over everything, but this is our cyber team at JetBlue.
[06:57.200 --> 07:01.780]  Uh, we have our leadership, you know, they control the money, they control the strategy, the directions.
[07:01.780 --> 07:03.020]  We have business partners.
[07:03.560 --> 07:05.480]  They, you know, help us for support.
[07:05.640 --> 07:06.740]  They host a lot of our tooling.
[07:07.200 --> 07:12.700]  Uh, we, we prioritize things in three major domains, risk, intelligence, and compliance.
[07:13.180 --> 07:17.280]  So risk that, you know, that's, that's architecture risk, how we design things.
[07:17.420 --> 07:21.160]  Is there risky, you know, ports open when we implement a new system?
[07:21.340 --> 07:24.140]  Uh, do people have, you know, overly permissive access?
[07:24.460 --> 07:27.840]  We have to whittle that down to reduce the risk, uh, from a design perspective.
[07:28.440 --> 07:33.040]  Uh, there's intelligence, you know, we, we, we, we collect, we collaborate a lot with other airlines.
[07:33.320 --> 07:34.460]  Uh, we have IR feeds.
[07:34.620 --> 07:36.620]  We do dark web scans for customer information.
[07:37.120 --> 07:40.120]  Uh, so we have a separate team that does all of our threat intelligence.
[07:40.480 --> 07:44.240]  And then we have compliance for all of the other regulatory bodies that have to audit us.
[07:44.480 --> 07:51.840]  So as an architect, you know, most of my work is integrations, design, consulting with the business when they have new systems, new software.
[07:52.360 --> 07:54.920]  And then Greg can speak a little bit about the day-to-day engineering ops.
[07:55.900 --> 07:56.360]  Yeah.
[07:56.520 --> 07:56.620]  Yeah.
[07:56.620 --> 08:03.660]  So on the operations team, um, our operations team is responsible for, you know, obviously maintaining the operation.
[08:04.060 --> 08:15.420]  Um, pretty much any changes, whether it's a firewall change or what have you, has to go through the operations team to make sure that, um, they're okay with it so they can approve it.
[08:15.420 --> 08:34.100]  Um, so, you know, it can get a little, it can get a little, um, wishy-washy when it comes to that because, uh, you know, if a change that a team needs is requiring to open like a ton of ports and, um, you know, they deem it to be unsafe, then it can get a little,
[08:34.220 --> 08:44.220]  you know, hectic, but, um, uh, other sort of things that the operations team does is just making sure that our tools don't break anything and get in the way of other, um, teams work.
[08:44.220 --> 08:59.960]  So, you know, we have some certain things that might, um, stop a program from running successfully and might just break the overall data flow or, um, you know, there are other examples, but, um, yeah, so, uh, and then from the engineering standpoint, what I do is,
[09:00.200 --> 09:02.120]  um, I build things, I break things.
[09:02.500 --> 09:08.700]  Um, Randy basically designs the, and architects the flow and then I will build that out.
[09:09.140 --> 09:16.920]  Um, right now we are pretty much finishing a major, major project of an sim that I won't name that we just implemented.
[09:17.320 --> 09:21.980]  Um, and I just finished building everything out basically for the rest of the team.
[09:22.140 --> 09:29.440]  And it's really cool to see our team now, um, interacting with it and using it and, and seeing how much it helps their, uh, you know, their jobs.
[09:30.840 --> 09:31.200]  Yeah.
[09:31.320 --> 09:33.200]  And I just thought of something, um, as he was talking.
[09:33.360 --> 09:40.680]  So like one like unique example, um, say a pilot gets a malware on their phone and they're sleeping and they have to fly the next day.
[09:40.840 --> 09:45.540]  We can't call them to wake them up because they need to have rest to be able to operate the plane.
[09:45.540 --> 09:52.100]  So there is specific FAA regulation, you know, you know, to, to, to protect that, uh, protect passenger safety.
[09:52.280 --> 09:53.940]  You don't want a tired pilot flying your plane.
[09:54.620 --> 09:59.180]  Um, so the way we solve that, you know, we have mobile EDR agents that we can access through remote shell.
[09:59.420 --> 10:08.020]  So if something happens on a device, we can remotely go in through the LTE connection, uh, segment the, kill the process, see what's happening, even wipe the device.
[10:08.200 --> 10:14.080]  Sometimes if they wipe the device when they get to the airport, they'll just give them a new device because we have desktop teams in all the airports.
[10:14.360 --> 10:19.440]  So that's one way we've kind of got around that, but that's kind of an example of what makes an airline unique.
[10:19.760 --> 10:22.460]  So there's things like that, uh, that happened all over it.
[10:22.920 --> 10:28.040]  So, so I'll take you through one big incident that happened a couple of years ago.
[10:28.320 --> 10:32.240]  Many of you were probably aware of it if you worked in any enterprise in the industry.
[10:32.420 --> 10:33.360]  So it's Log4j.
[10:33.780 --> 10:37.860]  So Log4j, uh, came out at 2021.
[10:38.580 --> 10:45.420]  Um, I was in Florida trying to have a vacation when this came out and I got looped in on calls for six days straight.
[10:46.360 --> 10:47.240]  So this is back...
[10:47.240 --> 10:49.220]  I just want to add, I wasn't on the team yet, so this is all right.
[10:49.960 --> 10:55.580]  Yeah, so I was an engineer at the time, Greg's position, and we first heard from our CDN provider.
[10:56.000 --> 11:00.820]  Uh, CDN is what copies your website out across the Internet so that it's fast in different regions.
[11:01.220 --> 11:04.580]  So when, you know, you see web requests, it hits the CDN first.
[11:04.760 --> 11:09.800]  So they first started seeing reports from other customers of, uh, these weird strings coming in.
[11:10.100 --> 11:15.080]  And the way the vulnerability works, the attacker can send the request with a JNDI string.
[11:15.220 --> 11:17.460]  It's a Java string that gets logged.
[11:17.680 --> 11:21.200]  Now, when it gets logged, it executes whatever is in that payload.
[11:21.520 --> 11:32.000]  So somebody could open a callback, they can execute code, they can open a reverse shell, and it'll happen without authentication and, you know, with full access.
[11:32.200 --> 11:36.000]  So that's a very dangerous, uh, vulnerability because you don't have to have access.
[11:36.140 --> 11:42.640]  You can do this remotely, and then you could target anybody that uses Java Log4j, which is like, you know, 80% of the Internet.
[11:43.280 --> 11:45.800]  So that's in a nutshell how the, this worked.
[11:45.980 --> 11:50.900]  So the first step, uh, when I was on the IR side, we looked at the vulnerability to try to understand it.
[11:51.020 --> 11:53.340]  This diagram shows you exactly how it works.
[11:53.580 --> 11:58.600]  And, you know, the first step is understanding, you know, how does the attacker actually get in and exploit.
[11:59.360 --> 12:04.960]  So I set up a vulnerable web app on one of our internal, uh, I think it was locally on my laptop.
[12:05.420 --> 12:11.020]  And, uh, this runs a web server with that vulnerable logging agent on it.
[12:11.740 --> 12:22.300]  Then I ran the exploit on myself locally and confirm that I can go from one port out to the other port, kind of replicating that same flow.
[12:22.780 --> 12:27.040]  And you can see here I have, it's kind of hard to read what, there's a callback at the bottom there.
[12:27.180 --> 12:28.460]  What you see there has coming back.
[12:28.760 --> 12:30.580]  So I was able to compromise myself.
[12:30.780 --> 12:32.400]  So now we knew this vulnerability is out there.
[12:32.540 --> 12:34.680]  It's real, you know, it's actually going to affect things.
[12:35.920 --> 12:37.660]  So then I looped in the red team.
[12:37.920 --> 12:39.780]  Uh, we were all on a major incident bridge.
[12:40.000 --> 12:44.880]  Uh, we looped in legal, our team, IT, and started the damage control.
[12:45.080 --> 12:48.240]  So we first had to enumerate, you know, what, where are we at risk?
[12:48.300 --> 12:56.000]  We have a website, we have a mobile app, we have third-party apps, we have internal apps, what's external, what's internal, and have them start testing it.
[12:56.200 --> 12:59.100]  So I'm in a conversation here on teams with our pen tester.
[12:59.440 --> 13:01.660]  Uh, thankfully he was in offshore hours.
[13:01.800 --> 13:04.060]  So even though this was late, he was up and working on it.
[13:04.380 --> 13:06.860]  And we started seeing payloads come in at this time.
[13:07.000 --> 13:12.820]  This is about two and a half hours after there was reports in the news and in the forums about this vulnerability.
[13:12.820 --> 13:19.000]  So I started freaking out seeing stuff come in and validating that if, if it's, it's him or is it someone else?
[13:20.380 --> 13:26.620]  Um, this was the playbook that I put together on the fly, just going through, you know, everything we have.
[13:26.780 --> 13:27.960]  So it was multiple vulnerabilities.
[13:28.240 --> 13:32.020]  Log4j was the most serious one at the top, but they had a few other ones.
[13:32.140 --> 13:34.380]  So you had to go through, you know, are we at risk?
[13:34.540 --> 13:35.200]  How does it work?
[13:35.620 --> 13:36.780]  Where are we at risk?
[13:36.900 --> 13:38.100]  And what do we do to fix it?
[13:38.360 --> 13:41.760]  So this shows, you know, kind of my methodology.
[13:41.760 --> 13:43.680]  Cause I was the incident leader on this bridge.
[13:44.300 --> 13:50.380]  And you can see each sheet in the spreadsheet has, uh, you know, a summary of all of that.
[13:50.540 --> 13:54.500]  And then the testing, threat hunting after the fact, and then how do we mitigate?
[13:54.840 --> 13:59.640]  So the mitigation, uh, we solved it with VCL.
[13:59.800 --> 14:02.620]  So on our CDN, uh, you can run code at the edge.
[14:02.720 --> 14:05.220]  So that's the first place an attacker hits when they're hitting us.
[14:05.360 --> 14:07.660]  So the best place to block it is the entry point.
[14:07.940 --> 14:10.200]  So Varnish was the language that we used.
[14:10.200 --> 14:14.660]  And I just put a pretty basic regex looking for the J and DI string in the middle.
[14:15.180 --> 14:17.300]  Uh, decoding it and blocking it.
[14:17.460 --> 14:18.540]  You get a 403 forbidden.
[14:18.940 --> 14:24.800]  And then, you know, at the same time in parallel, all of our teams on IT, we're rushing to update this library on any system.
[14:25.020 --> 14:29.440]  And it, the hardest part is, um, this library is big to third-party software too.
[14:29.440 --> 14:37.800]  So if you have like Tableau running on your computer, that's using this library, you got to reach out to Tableau now and tell them, Hey, you got to just got to update this and push out something.
[14:38.060 --> 14:39.540]  So that's where the bottleneck is.
[14:39.620 --> 14:44.060]  You can, we can control our own software, but getting the third parties to update was a pain.
[14:44.060 --> 14:49.940]  So it was, it was actually safer to just remove a lot of it off from public access while they were figuring out how to patch it.
[14:50.360 --> 14:53.700]  Um, so, and this is patching a third-party library and a third-party vendor.
[14:53.700 --> 14:58.340]  So it took a lot of time for a lot of these, uh, legacy ones to get their stuff updated.
[14:59.720 --> 15:05.040]  Um, as more people started hearing about the vulnerabilities, they started bypassing regex.
[15:05.400 --> 15:15.840]  And because of the WAF we had at that time was regex-based, I, this was, took up five days out of the six, was just constantly pushing out new VCL to get the new permutations of the attack.
[15:16.400 --> 15:23.660]  And good thing I know Python and I can use the APIs and I could code, cause I was just put like, we have 65 services on our CDN.
[15:23.800 --> 15:30.080]  If I had to go in manually and change this for each service every time, you know, by the time I change it, another one came out.
[15:30.200 --> 15:35.740]  So we had scripts that were talking to all of our services that can push this out, uh, pretty instantly.
[15:35.840 --> 15:38.640]  We got approval from the business to touch all these services at the same time.
[15:38.880 --> 15:45.520]  And we kind of kept up, uh, with all of the attacks until we got, uh, you know, a more intuitive solution in front of it.
[15:46.680 --> 15:52.400]  Uh, and speaking of that solution, Greg can kind of talk about the newer WAF that we have and how we're still seeing attacks from them.
[15:53.760 --> 15:53.960]  Yeah.
[15:54.060 --> 15:58.060]  So here's an example of it from, um, our, uh, WAF dashboard here.
[15:58.200 --> 16:01.340]  You can see pretty kind of easy to follow these.
[16:01.760 --> 16:08.800]  If you look at this site flagged IP suspected bad bot, these are tags that are automatically generated by the WAF.
[16:08.880 --> 16:14.440]  Um, if it sees behavior that it deems, you know, to be malicious, it will tag it with certain things.
[16:14.440 --> 16:18.640]  Um, and then of course you can see that it tagged it with the Log4j, JNDI.
[16:18.960 --> 16:23.660]  And then you have the exact payload that the, uh, this particular person used.
[16:24.040 --> 16:27.360]  Um, and then you look at the UA on the bottom right.
[16:27.700 --> 16:33.980]  Um, if any, are you familiar with, uh, user engines, you'll notice that this one is obviously malicious.
[16:34.240 --> 16:36.740]  It's got hacker1e plus salmon at the end of it.
[16:36.820 --> 16:38.300]  So obviously that's, that's nonsense.
[16:38.880 --> 16:41.760]  Um, and then, uh, what was I gonna say?
[16:41.760 --> 16:43.660]  Do you want, I'll move on to the next one.
[16:44.800 --> 16:52.900]  Um, and speaking of bot mitigation, I'm gonna go over a more recent example of how we kind of protect jetblue.com.
[16:53.400 --> 16:57.040]  Um, we recently, just this week, I've, um, actually, which is perfect.
[16:57.160 --> 16:58.040]  We had a fair sale.
[16:58.160 --> 17:03.480]  Um, it was our first percent discount based fair sale in like five or six years.
[17:03.480 --> 17:07.740]  So, um, we were expecting something like 13 to 15 million customers.
[17:08.080 --> 17:13.760]  Um, overall, we had 24.8 million flight searches over those two days.
[17:14.080 --> 17:17.440]  Um, 202.4 million hits on jetblue.com.
[17:17.580 --> 17:22.940]  Um, and it was generally like three times the amount of normal traffic we see on, um, any given day.
[17:23.080 --> 17:25.940]  So obviously, you know, it was, uh, it was a large, large increase.
[17:26.680 --> 17:37.540]  Um, and if we look at a closer look here at the data, um, this basically represents, uh, the purple line represents, um, allowed requests or two hundreds.
[17:37.940 --> 17:39.840]  Um, if you're familiar with the response codes.
[17:39.900 --> 17:45.520]  Um, and then the bottom line is pink and that represents, um, our block requests.
[17:46.140 --> 17:54.100]  Um, so overall we blocked 5.7 million, um, requests out of a total of 16.8.
[17:54.100 --> 17:55.900]  So, um, pretty good.
[17:57.280 --> 17:59.860]  Um, overall, honestly, it was a really smooth experience.
[18:00.180 --> 18:05.760]  I've been managing the WAF for a year and a half-ish now about.
[18:06.200 --> 18:10.560]  Um, and, uh, you know, it, it was, it was fantastic.
[18:10.900 --> 18:12.120]  There were no, really no issues.
[18:12.440 --> 18:14.720]  I always go into these fair sales expecting the worst.
[18:14.880 --> 18:19.300]  We've seen some really bad things before, but this was like, I really had to do nothing.
[18:19.560 --> 18:20.440]  Um, it was awesome.
[18:20.660 --> 18:23.980]  Um, and according to our digital products team, it was the smoothest sale in years.
[18:23.980 --> 18:37.660]  Um, you know, I've talked to the managers on that side and they've, um, you know, vented to me in the past before, you know, Randy and I, and some of the other people on our team really stepped in and took control of our web security.
[18:37.920 --> 18:46.600]  How, um, how much they were, they were suffering when it came to these sales, you know, there would be, um, bandwidth issues, latency issues.
[18:46.600 --> 18:49.920]  The site would, uh, crash for minutes at a time.
[18:50.140 --> 18:54.600]  And, um, you know, obviously this leads to loss of profit.
[18:54.720 --> 18:58.140]  You know, customers can't, um, you know, complete the checkout flow.
[18:58.600 --> 19:01.780]  They can't, um, they can't spend their money on a JetBlue flight.
[19:02.040 --> 19:05.340]  So, you know, leadership gets upset with that, et cetera, et cetera.
[19:05.580 --> 19:11.180]  So, um, site stability means more potential profit, which means everyone is happy.
[19:12.860 --> 19:14.460]  Uh, so how do we get to this point?
[19:14.560 --> 19:29.380]  You know, how do we go from, um, you know, having all these issues with bandwidth and latency, et cetera, to now where, um, you know, we're at a point where it's so smooth that we basically just have to monitor, um, at least for that last time.
[19:29.600 --> 19:30.040]  Knock on wood.
[19:30.420 --> 19:36.480]  Um, so basically it's a lot of lessons learned, um, from past instances and experiences.
[19:36.840 --> 19:54.480]  Um, every time we had a past event, or I'm sorry, past incident, um, you know, it requires a lot of work, um, a lot of brain power, a lot of meticulous investigation, um, into the data, into the logs, uh, what these bad actors are doing, how they exploited us.
[19:54.800 --> 19:59.300]  Um, in April, I believe, of this year, we had a DDoS attack.
[19:59.340 --> 20:03.180]  It was, like, on JetBlue.com at, like, 8 p.m.
[20:03.640 --> 20:10.660]  Um, and I believe it was, like, something like six billion requests and two minutes from thousands and thousands of IPs.
[20:10.900 --> 20:17.780]  Um, now, um, luckily we have a rate limiter on our CDN.
[20:18.040 --> 20:23.480]  Um, so that actually pretty much retracted all that bad traffic.
[20:23.660 --> 20:27.180]  So the site only went down for, like, a minute or two, which was amazing.
[20:27.340 --> 20:32.940]  Because in the past, before we had that rate limiter, the site would go down for, like, 30 to 40, 45 minutes at a time.
[20:32.940 --> 20:34.400]  Um, and obviously lose us money.
[20:34.720 --> 20:38.300]  Um, another example is GFS's Google Flight Search.
[20:38.500 --> 20:41.160]  I'm sure a lot of you guys are familiar with that or have even used it.
[20:41.500 --> 20:50.080]  Um, we came to notice that there was a lot of weird-looking traffic that was coming from Google Flight Search.
[20:50.280 --> 20:54.560]  So basically how it works, quick little flow, um, demonstration on the flow.
[20:54.840 --> 20:57.880]  Um, you go to Google Flight Search as a customer.
[20:57.880 --> 21:00.440]  You search for one of our flights, JetBlue's flights.
[21:00.440 --> 21:03.140]  Um, you hit submit on Google Flight Search.
[21:03.380 --> 21:04.680]  That calls one of our endpoints.
[21:04.820 --> 21:09.880]  So we were seeing a lot of, um, malicious traffic going towards that, um, that endpoint.
[21:10.320 --> 21:19.400]  Um, and when we did more research into it, uh, we came to find out that it was something like 95 to 98% of BOD traffic.
[21:19.880 --> 21:22.300]  Um, like, none of it was actually real traffic.
[21:22.440 --> 21:38.140]  So now we have, um, all these controls in place specifically for that endpoint, targeting, um, you know, uh, the things that these bad actors were, were, you know, doing in particular, um, and putting into their payloads.
[21:38.460 --> 21:43.520]  Um, and now, you know, it saved us a ton of bandwidth, because we were having a lot of issues with that.
[21:43.980 --> 21:57.980]  Um, and then the next part after that is establishing, you know, after you figure out what the bad data is, where it is, you know, how they're doing it, then you need to establish a creative solution, um, using our controls.
[21:57.980 --> 22:08.200]  So the WAP or the, um, you know, like Randy mentioned before, we can also use the CDN to do some creative solutions, like with regex or what have you.
[22:08.360 --> 22:16.020]  Um, we have to use these tools to, to create and, um, you know, combat these, these bad tactics that they're using.
[22:16.260 --> 22:19.500]  Um, and then after that, it's about implementing it all at the right place.
[22:19.500 --> 22:26.740]  You know, it's a lot of knowing your environments, your lower environments, your production environments, your all these things, and knowing all your endpoints.
[22:26.880 --> 22:27.900]  We have a ton of APIs.
[22:28.300 --> 22:32.960]  You have to know what each one is in the flow and how they're called and what calls what.
[22:33.100 --> 22:36.080]  And it's, it's important, um, to know all that stuff.
[22:36.240 --> 22:44.600]  And basically, as you can see from this diagram here, this little circle diagram, um, it pretty much creates a snowball effect is how I see it.
[22:44.680 --> 22:53.700]  Because the more we do this, the more it snowballs, uh, the more, you know, um, positive we get towards, you know, a better defense.
[22:54.160 --> 22:56.460]  Um, so yeah.
[22:56.900 --> 23:01.400]  Um, and then then for the future and, and, you know, building an even better defense.
[23:01.720 --> 23:06.620]  Um, you know, obviously we all know in this industry, things shift and evolve every day.
[23:06.780 --> 23:11.020]  Um, it's really important to stay on top of trends and the latest relevant vulnerabilities.
[23:11.020 --> 23:21.740]  You know, I'm myself, I'm sure like a lot of you guys, I'm always looking at the news and, um, seeing if there's any vulnerabilities that relate to JetBlue and, and the web apps we use.
[23:21.960 --> 23:23.760]  Um, cause mostly I'm on the website, obviously.
[23:24.720 --> 23:27.680]  Um, no matter what though, it will always be a game of trial and error.
[23:27.820 --> 23:29.600]  It's always in the same picture of whack-a-mole.
[23:29.720 --> 23:33.500]  You know, that's a lot of what, a lot of what this stuff comes down to on a daily basis.
[23:33.960 --> 23:40.480]  Um, which is now more of our analyst team is, is handling as Randy and I've been training them to take over the daily stuff.
[23:40.800 --> 23:44.920]  Um, so it's a lot of, you know, there's this new IP from Germany.
[23:45.160 --> 23:53.580]  That's, you know, slamming us and causing latency issues on JetBlue.com or whatever, you know, and then it's, you know, you have to think of a solution to block them.
[23:53.580 --> 23:55.400]  And then the next day there might be someone else.
[23:55.400 --> 23:56.380]  So it never ends.
[23:56.580 --> 23:57.840]  Um, it is a lot of work.
[23:58.200 --> 24:00.600]  Um, and then another big thing is collaboration.
[24:00.600 --> 24:06.560]  You know, we do a ton of collaboration with our DevOps team, our products team, um, et cetera, et cetera.
[24:06.560 --> 24:13.720]  We have a, a, uh, a huge thread, um, in teams where we ever almost every day, we're collaborating.
[24:13.760 --> 24:18.240]  We're all, um, you know, uh, posting our findings.
[24:18.240 --> 24:23.300]  If there's something malicious or suspicious looking, and we're always working together to, to solve the issue.
[24:23.420 --> 24:25.060]  And it's, it's really, it's actually really awesome.
[24:26.920 --> 24:27.360]  Okay.
[24:27.620 --> 24:28.100]  Back to Randy.
[24:28.520 --> 24:28.660]  Yep.
[24:28.800 --> 24:30.260]  And I'm building on what Craig said.
[24:30.500 --> 24:35.120]  Um, you know, if any of you in the room are also securing websites, you're securing the front end.
[24:35.120 --> 24:37.180]  Uh, you can always block things.
[24:37.380 --> 24:41.440]  In my opinion, as a security practitioner, it's much better to use deception.
[24:41.780 --> 24:42.640]  Confuse the attacker.
[24:43.000 --> 24:45.140]  Um, instead of whacking the mole, confuse the mole.
[24:45.580 --> 24:46.960]  You know, learn how to build honeypots.
[24:47.180 --> 24:48.640]  Learn how to build alias sites.
[24:48.960 --> 24:50.020]  Learn how to redirect traffic.
[24:50.160 --> 24:51.100]  Learn how to do tar pitting.
[24:51.580 --> 24:52.420]  Slow them down.
[24:52.860 --> 24:53.340]  Confuse them.
[24:53.420 --> 24:54.160]  Send them somewhere else.
[24:54.280 --> 24:59.180]  Send them to a, you know, a site where, you know, they can do all this stuff, but it's hitting a static page.
[24:59.180 --> 25:01.160]  So you have to use these techniques now.
[25:01.340 --> 25:02.760]  And you got to use them with automation.
[25:02.860 --> 25:04.760]  So you can't do things manually anymore.
[25:05.080 --> 25:07.540]  So we have a lot of automation on our side.
[25:07.660 --> 25:09.600]  Well, you know, most of our team is learning how to script.
[25:09.820 --> 25:11.340]  They're learning how to build things in containers.
[25:11.560 --> 25:12.380]  Deploy that at edge.
[25:12.840 --> 25:14.220]  Uh, even deploy it in JavaScript.
[25:14.500 --> 25:15.720]  And you can do all these things now.
[25:16.040 --> 25:18.740]  Uh, you know, since things are in the cloud, you can move quickly.
[25:19.280 --> 25:23.060]  Uh, so I, in my opinion, that's the best way to secure any public website.
[25:23.060 --> 25:32.520]  So now I'm going to shift a little bit and talk a little bit about how we deal with the government and some of the compliance regulations that we have to do and how we get logs off the planes.
[25:32.800 --> 25:35.700]  So three years ago, there was a major hack.
[25:35.840 --> 25:39.300]  Uh, you might've heard of it, uh, targeting a company called Colonial Pipeline.
[25:39.800 --> 25:47.180]  So they have the largest oil pipelines in the country from New Orleans to New York, and they got ransomware.
[25:47.300 --> 25:50.600]  So the whole pipeline shut down, all of their systems shut down.
[25:50.600 --> 25:54.600]  Uh, and this pipeline delivers jet fuel, delivers automotive fuel.
[25:54.940 --> 25:59.420]  It delivers, you know, raw diesel, all of, you know, major commodities throughout the country.
[25:59.460 --> 26:02.120]  Uh, and it shut down overnight when they got ransomware.
[26:02.320 --> 26:09.480]  They had to send the guy with a truck and an imager to each station in the pipeline from New Orleans to New York to get it back online.
[26:10.100 --> 26:13.860]  So, uh, I've, I don't know if they ever released who did it, but this happened.
[26:13.980 --> 26:15.680]  Gas prices went up a few years ago.
[26:15.680 --> 26:21.660]  And now in response to that, the Department of Homeland Security released numerous guidelines for critical infrastructure.
[26:22.020 --> 26:24.920]  Uh, in aviation, we have one called the ANSP.
[26:25.200 --> 26:26.940]  It's the Aircraft Network Security Program.
[26:27.440 --> 26:30.060]  And it's, you know, early in development.
[26:30.060 --> 26:31.680]  So there's been a lot of changes.
[26:32.080 --> 26:34.760]  Um, sometimes the government can be very challenging to work with.
[26:34.880 --> 26:37.080]  Um, they don't change for a while.
[26:37.200 --> 26:39.580]  And then all of a sudden they'll change all the regulations in a week.
[26:39.580 --> 26:41.680]  And then you have to keep up with them or you get fined.
[26:42.260 --> 26:44.260]  And some of these fines can be a quarter of a million dollars.
[26:44.760 --> 26:45.040]  So...
[26:45.600 --> 26:51.520]  One of the requirements we have to do is when a plane is in maintenance, we have to retrieve logs off of that plane.
[26:52.040 --> 26:55.600]  Um, we have Airbus A320s, uh, 220s.
[26:55.700 --> 26:57.040]  We have Embraer 190s.
[26:57.420 --> 27:05.240]  Um, as these come in, when they're refueling or when techs are in, uh, they go into the planes and they take off, uh, the logs from the planes.
[27:05.240 --> 27:07.520]  Uh, with, you know, some of them use floppy disks.
[27:07.620 --> 27:10.620]  Some of the older planes still have floppy disks, um, or serial connections.
[27:10.840 --> 27:11.860]  Some of the newer ones have USB.
[27:12.360 --> 27:18.500]  But they have to pull out, like, 15 floppy disks, put them together, get the data into something that's, like, Unicode or something readable.
[27:19.040 --> 27:20.160]  And then put it into our NAS.
[27:20.340 --> 27:24.440]  And Greg, um, you know, I handle that part of it, working with tech ops and getting that system in place.
[27:24.820 --> 27:26.380]  It's still a sneaker net, but it works.
[27:26.800 --> 27:34.960]  And then once that data is in, you know, data that a computer can read, um, Greg has done a lot of work with our logging platform that he can share about how we get that login to our SIM.
[27:34.960 --> 27:39.220]  Um, so basically just a quick overview of this.
[27:39.380 --> 27:39.460]  Yeah.
[27:39.580 --> 27:46.160]  So as Randy was saying, um, the tech ops, uh, tech ops maintenance technician will, um, be on the plane.
[27:46.160 --> 27:49.180]  They'll, they'll put the logs on the USB stick.
[27:49.300 --> 27:56.160]  They'll go back to the office, to their desktop or laptop or whatever, um, and, and upload that to, to our NFS.
[27:56.160 --> 28:10.920]  Um, from there, uh, our SFTP server will, will take that, um, those, those log files, that batch of logs and they'll, um, that, that SFTP server will, will forward that to, um, cloud storage.
[28:11.640 --> 28:18.180]  And then from the cloud storage, um, I will pull that with, um, a log pastor that we're using.
[28:18.460 --> 28:22.280]  Um, and then that log pastor will, um, throw it into our SIM.
[28:22.500 --> 28:25.280]  And then from, um, actually that's where I am right now.
[28:25.420 --> 28:29.200]  I just got some data into our SIM from these, from these air aircraft logs.
[28:29.200 --> 28:43.960]  And I'm working with our detection engineering team now to parse the logs out, um, get the data, uh, formatting to the place where it should be so that, um, you know, our detection team can, um, write, um, efficient detections.
[28:46.000 --> 28:46.500]  Yeah.
[28:46.580 --> 28:51.100]  And as we discover things, we're partnering with Airbus, essentially all the systems on a plane are Linux.
[28:51.100 --> 28:56.200]  So, you know, wherever we find issues, we can implement controls with the OEM, Airbus.
[28:56.420 --> 29:07.560]  We can control process, memory, open ports, uh, disk access to all these different subsystems that control things like in-flight entertainment, in-flight Wi-Fi, avionics, the toilet, whatever you want on the plane.
[29:07.980 --> 29:10.080]  We can detect it, uh, once we get these logs.
[29:10.640 --> 29:14.000]  Thankfully, these systems are pretty air-gapped, so we don't really see much.
[29:14.220 --> 29:17.560]  But, you know, if we see something, we have to report it to the federal agency.
[29:17.560 --> 29:22.580]  Then we have to work with Airbus and the government, uh, to figure out what happened, why it happened.
[29:22.740 --> 29:23.480]  Did it cause impact?
[29:23.740 --> 29:25.440]  Did it potentially compromise the aircraft?
[29:25.620 --> 29:26.660]  Do you have to take it out of service?
[29:27.300 --> 29:28.680]  So that's where we are today.
[29:28.780 --> 29:31.220]  And every airline, uh, in the U.S.
[29:31.360 --> 29:33.320]  has to, you know, abide by these same standards.
[29:33.520 --> 29:37.520]  So it's an emerging kind of field, like a subfield for logging and detection.
[29:37.900 --> 29:40.940]  But it's very interesting because you see different things that you wouldn't see.
[29:41.060 --> 29:43.780]  You know, you see stuff from the engine, you see stuff from the radar systems.
[29:44.220 --> 29:48.020]  So it's Linux systems, but it's a lot more, it's a lot cooler logs than you would normally see.
[29:48.220 --> 29:50.720]  We have to, we have to keep these logs for like at least a year.
[29:50.860 --> 29:52.000]  We can get in a lot of trouble, basically.
[29:52.360 --> 29:52.500]  Yeah.
[29:52.540 --> 29:52.820]  Yeah, so.
[29:54.480 --> 29:59.780]  So that leads me to my, you know, our last part of the talk, which is the AI SAC.
[29:59.980 --> 30:05.020]  So most every industry has an ISAC, which stands for information sharing and collaboration.
[30:05.540 --> 30:08.180]  Um, the one for aviation is aviation ISAC.
[30:08.180 --> 30:11.160]  So there's a financial ISAC, the health ISAC.
[30:11.420 --> 30:17.800]  Um, if you're a member of a company that's in these industries, I encourage you to talk to your board, talk to your leadership to get you membership.
[30:18.020 --> 30:21.200]  And it costs a lot, but the information you get is valuable.
[30:21.860 --> 30:24.840]  So this is a kind of a blurb of the aviation ISAC.
[30:24.980 --> 30:26.060]  You can read about them online.
[30:26.220 --> 30:26.620]  They're public.
[30:27.060 --> 30:29.240]  Um, and they do exactly what I just went through.
[30:30.180 --> 30:33.180]  And this is an example of a working group that we have.
[30:33.360 --> 30:37.980]  So we meet every quarter with different airlines, different airports, different manufacturers.
[30:38.080 --> 30:40.180]  And we talk through different problems we have.
[30:40.380 --> 30:43.400]  I'm a member of the, uh, network security architecture working group.
[30:43.580 --> 30:45.800]  Um, you can see the bottom left corner of the slide.
[30:45.940 --> 30:47.040]  This is TLP green.
[30:47.260 --> 30:49.060]  It's traffic light protocol, meaning green.
[30:49.060 --> 30:49.760]  I could share it.
[30:49.940 --> 30:52.480]  Um, it's an intelligence, you know, quarterly.
[30:52.740 --> 30:55.120]  So if it's red or amber or something, we can't share it.
[30:55.520 --> 30:57.240]  So this is just a generic diagram.
[30:57.240 --> 31:00.100]  None of this is real, but this is a typical airline network.
[31:00.440 --> 31:07.740]  You know, you have sat comms, you have local networks, you have wide networks, you have aircraft networks and user networks, VPN networks, clouds, data centers.
[31:07.960 --> 31:08.500]  So it's large.
[31:08.840 --> 31:15.780]  You know, I think, you know, we have our network alone has thousands of subnets, hundreds of thousands of devices, 60,000 people.
[31:16.100 --> 31:21.220]  So the networking in an airline is, you know, very challenging to work with.
[31:21.480 --> 31:24.200]  And it's always moving and it's never supposed to shut off.
[31:24.200 --> 31:27.500]  So those are some of the challenges we discuss in this group.
[31:27.500 --> 31:33.340]  And there's a lot of talented people that I've met from United, Delta, Boeing, Airbus, um, in these groups.
[31:33.380 --> 31:34.980]  And we solve those challenges together.
[31:35.660 --> 31:38.300]  Um, we also went to Southwest last year.
[31:38.620 --> 31:43.160]  Um, surprisingly, they have a pretty big cyber team, almost 300 people, uh, on their team.
[31:43.160 --> 31:47.800]  Um, and they gave a pretty good overview of how they run cyber in their business.
[31:48.040 --> 31:50.320]  They shared everything, you know, in this closed forum.
[31:50.720 --> 31:53.240]  Um, and you can see some of the topics here on the left.
[31:53.320 --> 31:55.060]  And then we got some nice stuff from them too.
[31:55.600 --> 32:02.140]  Um, and then we got to see, we got to work with, um, Airbus to kind of understand more about how the actual planes work.
[32:02.140 --> 32:03.540]  Like, you know, I have an IT background.
[32:03.820 --> 32:05.040]  I don't have an OT background.
[32:05.280 --> 32:06.820]  So I had no idea.
[32:06.960 --> 32:08.140]  Like I know TCP IP.
[32:08.400 --> 32:09.280]  I was a network engineer.
[32:09.820 --> 32:10.980]  I don't know what ARNIC is.
[32:11.080 --> 32:14.420]  I don't know what, you know, TCAS is, what AIDs are, what GNAS is.
[32:14.560 --> 32:18.620]  Like these are all non-TCP protocols that, you know, we have to learn.
[32:18.740 --> 32:19.340]  How do they work?
[32:19.580 --> 32:20.820]  You know, how are they segmented?
[32:21.240 --> 32:22.620]  You know, safety is our number one value.
[32:22.740 --> 32:24.540]  If this is compromised, is the plane compromised?
[32:24.720 --> 32:27.080]  What happens to the pilot if he sees the false reading?
[32:27.080 --> 32:31.920]  So all of these things we discuss with the manufacturers to kind of understand where the risks are.
[32:32.080 --> 32:33.940]  And then we can quantize the risks and address them.
[32:34.580 --> 32:40.360]  So, and if anyone's, you know, old enough, they might recognize the little blow-up guy on the left from the movie, Airplane.
[32:40.620 --> 32:45.160]  So they always bring him and he's our, you know, pen tester when we're testing the avionics.
[32:46.220 --> 32:48.020]  And it's a good movie if you haven't seen it.
[32:48.720 --> 32:51.520]  Um, so this is in Southwest's hangars.
[32:51.740 --> 32:53.820]  We literally got to see how they take the plane apart.
[32:53.820 --> 32:58.180]  Uh, physically, you know, as a hacker, you know, where can I go in the plane?
[32:58.300 --> 32:58.940]  What can I touch?
[32:59.360 --> 33:00.740]  You know, is this serial port live?
[33:00.900 --> 33:01.480]  Can I plug in?
[33:01.600 --> 33:03.040]  What information can I get from it?
[33:03.640 --> 33:04.740]  Um, is it even a serial port?
[33:04.840 --> 33:06.180]  Is it some other kind of interface?
[33:06.540 --> 33:08.320]  So that was really fascinating.
[33:08.600 --> 33:11.720]  Um, like learning what happens in the cabin when you go in the plane?
[33:12.220 --> 33:16.400]  Um, where the risks are, uh, you know, what's different from plane to plane?
[33:16.700 --> 33:22.780]  And then you can see in the top, uh, right picture, these are people from Airbus, Boeing, Lufthansa.
[33:22.780 --> 33:25.960]  Uh, so the whole world comes together at these forums, which is great.
[33:26.320 --> 33:29.060]  Um, and then at the bottom right is our threat and tell team.
[33:29.340 --> 33:31.780]  Uh, so we got to meet cause all of us are remote now.
[33:32.100 --> 33:34.520]  Uh, when we went to this, I met some of them for the first time.
[33:34.520 --> 33:37.280]  Um, and then this was cool.
[33:37.400 --> 33:39.560]  We partnered with Embraer Riddle, uh, university.
[33:39.620 --> 33:46.120]  So we have a lot of students, you know, not many people study IT and avionics at the same time.
[33:46.280 --> 33:47.760]  Usually we have to look for people in the Navy.
[33:47.760 --> 33:56.180]  We're trying to bridge the gap to get students in, you know, learning how planes operate and how IT systems operate so they can secure planes in the future.
[33:56.440 --> 33:58.860]  And what this university does, they do a hackathon every year.
[33:58.860 --> 34:08.420]  They have little model planes with Raspberry Pis that control, uh, flaps, motors, cockpit, uh, all an emulated software, but the software is from like Airbus or Boeing.
[34:08.580 --> 34:09.520]  So it looks like it's real.
[34:09.740 --> 34:14.060]  And then they can connect to it over a LAN port and they can start, you know, learning.
[34:14.060 --> 34:15.300]  What does this command do?
[34:15.560 --> 34:16.760]  Does it move the flap?
[34:16.940 --> 34:19.180]  Does it open, you know, this bus on the cockpit?
[34:19.760 --> 34:21.460]  Uh, how can we interact with it?
[34:21.620 --> 34:23.740]  And then it just gets them interested.
[34:23.980 --> 34:29.040]  So maybe, you know, they'll go forward, get a degree at the university, also study IT at the same time.
[34:29.140 --> 34:29.940]  They can do a dual degree.
[34:30.040 --> 34:32.200]  And at the end of that, they're ready to come into the industry.
[34:33.640 --> 34:34.920]  So that's the end of our talk.
[34:35.060 --> 34:36.180]  I'll leave some time for questions.
[34:36.360 --> 34:40.300]  If anyone has questions, you know, feel free to come up or raise your hand or shout it up.
[34:44.060 --> 34:44.320]  Thank you.
[34:45.040 --> 34:45.680]  Thank you.
[34:46.360 --> 34:47.040]  Thank you.
[34:47.340 --> 34:48.040]  Thank you.
[34:51.980 --> 34:53.040]  You got a question.
[34:53.360 --> 34:53.720]  Sure.
[34:53.780 --> 34:54.580]  I don't know if this works.
[34:54.760 --> 34:55.140]  We can hear you.
[34:55.460 --> 34:55.480]  Yeah.
[34:56.400 --> 34:56.740]  Okay.
[34:56.760 --> 34:57.580]  I'll just talk loud.
[34:57.920 --> 35:04.220]  So there were a couple of things on the org diagram that I didn't actually see that would normally be in a security program.
[35:04.400 --> 35:06.180]  So I'm just wondering whether you guys have them.
[35:06.820 --> 35:07.000]  Yeah.
[35:07.060 --> 35:10.300]  The first is AppSec testing functions.
[35:10.600 --> 35:14.580]  So things like static analysis, dynamic analysis, software analysis.
[35:15.780 --> 35:22.720]  And then the other question I had is, do you guys have a security champions program or a security training program?
[35:23.160 --> 35:24.740]  And if you do, how did those work?
[35:24.980 --> 35:25.260]  Yeah.
[35:25.480 --> 35:28.540]  I couldn't fit everything in the little boxes, but we do have an AppSec.
[35:28.800 --> 35:30.880]  They are between our vulnerability management.
[35:31.440 --> 35:34.020]  Pen testing reports to the vulnerability management team.
[35:34.020 --> 35:36.520]  So they kind of operate cycles there.
[35:36.680 --> 35:38.700]  We do static dynamic analysis as well.
[35:38.960 --> 35:41.100]  We have a bug bounty program that just launched.
[35:41.260 --> 35:45.660]  So if you find bugs, we're trying to get more rewards out there like miles or tickets eventually.
[35:46.060 --> 35:48.700]  But we have a HackerOne presence as well.
[35:48.800 --> 35:50.560]  If you find stuff, you can submit it there.
[35:50.820 --> 35:52.140]  And we do a pretty good job.
[35:52.260 --> 35:57.560]  I think that team is very talented and they do a pretty good job of operating and resolving those.
[35:57.560 --> 36:03.040]  And they operate on U.S. time and overseas time so that even if there's stuff overnight, they can address that.
[36:03.700 --> 36:05.480]  And we do have a champions program.
[36:05.480 --> 36:07.040]  So we work a lot with IT.
[36:07.580 --> 36:12.320]  You know, we have system engineers, network engineers that we partner with that we need to...
[36:12.320 --> 36:16.900]  You know, if a major incident happens, I have people on my phone, I can call right away, get them on a bridge.
[36:17.340 --> 36:20.300]  If we see issues, you know, in our clouds and our data centers.
[36:20.840 --> 36:27.580]  And then we have partners on the outside federally and, you know, with Airbus, with different systems that we use.
[36:28.260 --> 36:29.700]  And we can reach them pretty quickly.
[36:29.940 --> 36:33.220]  And then if they see issues, they can also reach out to us in retrospect.
[36:36.240 --> 36:36.640]  Training.
[36:36.780 --> 36:40.000]  We have training under the vulnerability team.
[36:40.000 --> 36:41.840]  They run, like, end user training.
[36:42.260 --> 36:44.240]  We have phishing tests all the time.
[36:44.860 --> 36:47.240]  We test our, you know, mobile workforce.
[36:47.600 --> 36:51.740]  The pilot, the ones who are not in the office day to day, who maybe are not exposed to technology much.
[36:51.980 --> 36:53.780]  We do a lot of phishing testing with them.
[36:54.400 --> 36:59.840]  Now we're doing phishing testing because now you can emulate a voice now, which is even scarier with AI.
[37:00.280 --> 37:01.420]  So we're doing things like that.
[37:01.560 --> 37:04.900]  We're training VIPs as well to recognize impersonation.
[37:05.940 --> 37:07.180]  Those are heavily targeted.
[37:07.180 --> 37:10.500]  And then we have general training for IT for our developers.
[37:11.740 --> 37:16.060]  You know, we give them co-training to make sure they're doing secure code, secure design.
[37:16.840 --> 37:19.660]  And then we have detections to see if they're actually listening to us.
[37:22.800 --> 37:23.700]  Any other questions?
[37:30.510 --> 37:30.990]  Yeah.
[37:31.230 --> 37:36.090]  For the DDoS attack, we found reports on the dark web that it was a state actor.
[37:36.730 --> 37:41.910]  I think it was related to the ongoing war in the Middle East.
[37:42.090 --> 37:44.230]  So we're a partner with El Al.
[37:44.630 --> 37:46.130]  And El Al is an Israeli airline.
[37:46.130 --> 37:50.890]  I think because we're partners, they were attacking that airline and any partners that they had.
[37:51.010 --> 37:54.630]  So we were just caught in the crossfires there because that was a huge botnet.
[37:55.090 --> 37:58.310]  That was, you know, terabytes hit us in an hour.
[37:58.310 --> 37:59.770]  So it was crazy.
[37:59.970 --> 38:08.570]  Also, um, last year, last two years, we've had, um, reattempted attacks from, uh, this threat actor.
[38:09.070 --> 38:14.710]  Um, I think they were based in Africa and they just, they're known for targeting the aviation industry.
[38:14.710 --> 38:17.810]  And we, we kept seeing, um, malicious stuff from them.
[38:17.990 --> 38:19.910]  Don't really think they're doing it much anymore.
[38:20.130 --> 38:21.650]  Not sure if I haven't really heard much, but yeah.
[38:21.810 --> 38:32.090]  So we do know some, obviously it's hard to tell sometimes, but yeah, we, we have a good idea of some, some of these threat actors that, um, that will, um, hit us, uh, continuously, so.
[38:32.470 --> 38:32.790]  Yeah.
[38:35.030 --> 38:37.890]  Bug bounties, and I've, I've helped run a bug bounty program.
[38:38.150 --> 38:41.530]  I was wondering how the FAA figures into your bug bounty scope.
[38:41.530 --> 38:45.630]  Cause that can be really sensitive for hackers for certain telecommunications, right?
[38:46.250 --> 38:46.550]  Yeah.
[38:46.950 --> 38:53.410]  They have an open, so that would go through, like, if you wanna, you know, do a bounty on the planes or something, you would have to work with the manufacturer.
[38:53.850 --> 38:55.970]  So, like, Airbus has their own bug bounty program.
[38:55.970 --> 38:59.170]  We have our own as an operator, mostly for web stuff.
[38:59.510 --> 39:01.830]  But, um, Boeing actually has a really good one.
[39:02.290 --> 39:04.430]  Um, they have a good cybersecurity bug bounty program.
[39:04.810 --> 39:05.890]  They pay a lot more than we can.
[39:06.670 --> 39:08.250]  And, you know, you can work with them.
[39:08.390 --> 39:10.550]  I don't know if the government has, like, FAA.
[39:10.550 --> 39:11.810]  I know CESA probably does.
[39:12.450 --> 39:14.150]  Uh, DHS is probably gonna launch something.
[39:14.370 --> 39:18.810]  But, uh, we would work in conjunction with the agencies, um, for something like that.
[39:18.930 --> 39:21.690]  But, uh, as far as our program, it's mostly geared towards the website.
[39:22.110 --> 39:24.850]  Uh, anything on the equipment would go through the manufacturer.
[39:26.570 --> 39:26.970]  Thanks.
[39:27.250 --> 39:27.410]  Yeah.
[39:28.970 --> 39:31.230]  Um, so a couple questions.
[39:31.750 --> 39:37.630]  Uh, firstly, I know you guys said that a lot of those aircraft systems are air-gapped, so you see lower volume of threats.
[39:37.810 --> 39:42.830]  I was just wondering if you could speak into, uh, you know, what you do see, what sorts of threats and attacks.
[39:43.050 --> 39:48.550]  And, uh, second question, you know, what brought you, both of you, to work at an airline in particular?
[39:49.150 --> 39:49.290]  Yep.
[39:49.290 --> 39:51.410]  Um, so I can, I've seen some of the logs.
[39:51.650 --> 39:54.310]  Um, you know, a lot of it is false positives right now.
[39:54.630 --> 39:58.050]  Um, if a system fails or a disk fails, it might look like something else.
[39:58.170 --> 40:04.750]  If it starts re-encoding files, our sim might think it's, uh, ransomware, where it's just updating and changing all the extensions.
[40:05.090 --> 40:07.530]  Because these, the software on these planes are ancient.
[40:07.990 --> 40:09.890]  Um, the stuff that they run are ancient.
[40:09.890 --> 40:13.590]  So, it's, a lot of it is kind of looking, you know, is this normal?
[40:13.730 --> 40:14.650]  And collaborating with Airbus.
[40:14.830 --> 40:19.530]  Airbus has given us a lot of good runbooks to explain why their software is built this way.
[40:19.750 --> 40:20.910]  But in a way, that's good.
[40:21.030 --> 40:25.810]  It's kind of like obfuscation by design, because even if someone gets in, how does this work?
[40:25.990 --> 40:28.630]  It's, it's so ancient, you know, not a lot of people know anymore.
[40:28.790 --> 40:35.570]  And then some of the protocols that they use, especially on the avionics side, you really have a small body of knowledge that could dissect that.
[40:35.690 --> 40:41.010]  And it's closed knowledge, so it's not very shared, unless you, you know, have a military background or you've been in avionics tech.
[40:41.490 --> 40:44.810]  So, thankfully, you know, those two factors keep them pretty secure.
[40:44.990 --> 40:48.930]  And, um, I joined the airline actually right before COVID.
[40:49.310 --> 40:51.290]  I, I came from retail, which was falling apart.
[40:51.430 --> 40:53.410]  I thought I'd join an airline and then we'd get a pandemic.
[40:54.030 --> 40:56.690]  Uh, but, you know, I live very close to JFK.
[40:56.930 --> 40:57.810]  I love to travel.
[40:58.050 --> 40:59.430]  I'm up to 73 countries now.
[40:59.930 --> 41:06.450]  Um, you know, it doesn't pay as much as other industries, but if you love planes, you love traveling, uh, it's a good place to be.
[41:06.450 --> 41:07.890]  And I think probably the same for you.
[41:08.630 --> 41:09.750]  Yeah, pretty much the same for me.
[41:09.850 --> 41:10.610]  I love to travel.
[41:10.950 --> 41:12.910]  Um, you know, always liked Jeff Blue.
[41:13.190 --> 41:14.530]  We're a pretty reputable airline.
[41:14.730 --> 41:19.830]  So it felt like, you know, as I was trying to kind of pivot into tech, it felt like a good place to be.
[41:19.950 --> 41:24.930]  And now, you know, all this time later, I'm glad I did it because, um, I've gotten amazing experience.
[41:25.110 --> 41:30.550]  Um, as Randy also touched on earlier, our network is incredibly complicated and just our environment in general.
[41:30.550 --> 41:43.530]  So I feel like the things that we deal with, um, you know, when it is time to go to that next step of the journey to another company, whatever, I'll be like, I've seen a lot already, so I can probably maybe handle it depending on how good the company is.
[41:43.530 --> 41:45.570]  But yeah, so...
[41:45.570 --> 41:46.270]  Awesome.
[41:46.390 --> 41:46.510]  Thanks.
[41:50.050 --> 41:50.870]  Any other question?
[41:52.550 --> 41:52.990]  Yeah.
[41:58.720 --> 41:59.160]  Yeah.
[41:59.560 --> 41:59.840]  Otto.
[42:00.980 --> 42:01.740]  It's a good movie.
[42:01.880 --> 42:02.500]  One question there.
[42:05.560 --> 42:09.400]  Uh, when I started, we had like, maybe seven or eight.
[42:09.660 --> 42:13.180]  It was very... I have one right here in the audience who used to be a party.
[42:14.160 --> 42:15.500]  Um, it was bad.
[42:15.580 --> 42:17.540]  It was hard because we weren't funded well.
[42:17.920 --> 42:20.120]  Um, you know, cyber was like an afterthought.
[42:20.120 --> 42:31.240]  Uh, when we started, but in the last few years, especially since there's been regulatory governance now, uh, we were about 35 people, which is still small for the size of the company.
[42:31.440 --> 42:35.620]  Like Southwest has almost 300, you know, but they're a much bigger airline.
[42:35.780 --> 42:38.900]  JetBlue actually is only like eight or 9% of the U.S.
[42:39.060 --> 42:39.240]  market.
[42:39.760 --> 42:44.500]  Um, the other four big ones, Delta, Southwest, United American control like 80%.
[42:44.500 --> 42:46.060]  So they have much bigger teams.
[42:46.440 --> 42:48.460]  Um, but we do a lot with automation.
[42:48.460 --> 42:50.640]  You know, we don't do things manually here.
[42:51.280 --> 42:53.720]  Um, and we use a lot of good technologies.
[42:53.720 --> 42:56.520]  You know, we have, you know, a lot of AI systems inside.
[42:56.720 --> 42:57.880]  We have cloud systems.
[42:58.040 --> 43:00.080]  A lot of these other carriers are still in data centers.
[43:00.900 --> 43:05.140]  So because of that, you know, we're able to control more with less people and do it faster.
[43:06.620 --> 43:07.180]  Yep.
[43:08.600 --> 43:09.160]  Cool.
[43:09.360 --> 43:10.140]  So I've got one.
[43:10.360 --> 43:17.980]  Um, you mentioned earlier, you don't encounter a lot of folks who are studying avionics and cybersecurity at the same time.
[43:18.260 --> 43:33.380]  Um, for anybody who's currently a student or if somebody has a background in cybersecurity and they're interested in joining your part of the industry, what would you recommend to them in order to get their foot in the door or introduce themselves or, or what they should, should study?
[43:36.120 --> 43:40.180]  Yeah, I would start learning, you know, how plain systems work.
[43:40.440 --> 43:41.520]  Learn, you have to know Linux.
[43:41.760 --> 43:42.860]  All of these systems are Linux.
[43:43.440 --> 43:45.520]  Um, they're very old flavors of Linux.
[43:45.780 --> 43:51.980]  So you have to kind of go back and learn commands you might not use on a modern system like Rocky Linux or something.
[43:51.980 --> 43:54.060]  These are systems that are from the nineties.
[43:54.500 --> 44:00.400]  So, um, learning about different protocols besides TCP IP, um, you can Google a lot of it.
[44:00.620 --> 44:06.720]  Uh, there's a lot of stuff from the Navy, a lot of stuff from the manufacturers that have white papers, uh, but it's a lot to decipher.
[44:07.080 --> 44:12.840]  So if you want, you know, find me after the talk, I could introduce you to this guy here in the middle.
[44:12.960 --> 44:18.940]  I don't know if you could see him with the mouse, but he's the professor who runs their cyber avionics program at Embraer-Riddle.
[44:18.940 --> 44:21.660]  And he's looking for students, you know, there's a gap.
[44:21.880 --> 44:23.540]  There's not a lot of people studying this stuff.
[44:24.240 --> 44:26.700]  Um, so I could definitely connect you with him.
[44:27.100 --> 44:29.440]  Um, you might have to go to one of their campuses though.
[44:29.540 --> 44:30.740]  I don't know if they do remote stuff.
[44:31.000 --> 44:34.940]  Uh, they have campuses in Florida and Arizona and he would be the best resource.
[44:37.440 --> 44:43.620]  That, just to add to that real quick too, even that, I don't mean you were talking about, uh, you know, we were talking about DEFCON or whatever.
[44:43.820 --> 44:48.540]  Um, uh, they have like the, they had the whole aviation village there too, right?
[44:48.540 --> 44:59.000]  And there's a lot of great people that if someone was interested in getting into it, you know, they could even do something like that and, and talk to the people there and work, uh, network and all that, all that fun stuff to, to get a leg up on everything.
[44:59.260 --> 44:59.540]  Yeah.
[44:59.620 --> 45:02.020]  If you haven't been to DEFCON, go to aviation village there.
[45:02.140 --> 45:03.480]  They have them in a few conferences now.
[45:03.900 --> 45:04.840]  Uh, and they're great.
[45:05.120 --> 45:05.260]  Yeah.
[45:07.320 --> 45:12.500]  So you mentioned that you use, uh, the cloud a lot more than some of your competitors.
[45:12.760 --> 45:17.580]  And that's giving you advantages in terms of, uh, labor and, um, flexibility.
[45:17.900 --> 45:20.740]  Have you found any disadvantages of going that way?
[45:20.920 --> 45:25.000]  Like in terms of expense or reliability or anything else?
[45:25.000 --> 45:25.580]  Yeah.
[45:25.860 --> 45:25.920]  Yeah.
[45:26.080 --> 45:31.160]  Like, um, I w I recently had to back up all of our old SIMS data for one year.
[45:31.200 --> 45:33.180]  It came out to 1.6 petabytes.
[45:33.560 --> 45:35.180]  I didn't realize that at the time.
[45:35.320 --> 45:42.360]  And we got hit with a large bill, uh, from our provider and from the cloud, which was also hosted in the same network.
[45:42.580 --> 45:53.680]  Uh, thankfully we were able to arbitrate the bill down, but there's unexpected costs when you do things that run like in the background or on, you know, the back channels that you won't see until you see the bill.
[45:53.980 --> 45:58.180]  So, uh, forecasting is really important from a cyber perspective for things like that.
[45:58.280 --> 46:03.400]  Cause we run with a lot of dynamics, you know, we could spin up containers and they can easily go into thousands of containers.
[46:03.760 --> 46:05.840]  Uh, and there's costs associated with that.
[46:06.060 --> 46:12.760]  So, uh, once that happened, we've kind of set budgets now to get alerts for those kinds of things, but the speed of it makes up for it.
[46:12.760 --> 46:22.000]  So, you know, if I had to do that on a NAS or, you know, back up all of that stuff in the data center, you know, I'd have to wait, you know, weeks for some of that stuff to transfer.
[46:22.160 --> 46:25.900]  And now I can use our provider's backbone to move that data pretty quick on the cloud.
[46:26.060 --> 46:32.080]  And even speaking about our NAS, even that now, our, our, our platform engineering team is migrating all that to the cloud as well, you know?
[46:32.380 --> 46:35.420]  So we're really trying to, to get as much up there as we can, you know?
[46:35.960 --> 46:36.320]  Yeah.
[46:36.400 --> 46:45.500]  Our goal is to get out of data centers, um, you know, everything now you can do from an API, you can do from a browser, you can manage your workforce, you can run business apps.
[46:45.820 --> 46:49.900]  There's not really a need for data center anymore unless you need it for some compliance reason.
[46:50.140 --> 46:50.760]  Thank you.
[46:50.760 --> 46:51.580]  Thank you everyone.
[46:51.580 --> 46:51.620]  Thank you everyone.
[46:51.900 --> 46:52.220]  Yeah.
[46:52.220 --> 46:52.420]  And this is the second one.