[00:00.000 --> 00:04.020]  And he's here to talk about some of the interesting vulnerabilities that have come up through SSL.
[00:04.240 --> 00:05.080]  Please welcome Jimmy O.
[00:11.810 --> 00:12.410]  Thank you.
[00:12.990 --> 00:13.990]  Can everybody hear me all right?
[00:15.250 --> 00:17.450]  Yeah, is this house music going to keep going through the talk?
[00:18.750 --> 00:19.810]  I'm cool with it, it can.
[00:21.690 --> 00:22.510]  So yeah, hi.
[00:22.690 --> 00:26.450]  I'm Jim O'Leary, affectionately known to some as Jimmy O.
[00:26.930 --> 00:29.230]  I work at Twitter, and it's an honor.
[00:29.450 --> 00:40.990]  I'm very humbled to have the chance to talk to you, hopefully, educate and entertain with some of the things we've learned as we went from rolling out a widespread SSL deployment at Twitter.
[00:42.310 --> 00:45.670]  So it's been a bit of a bumpy ride lately for SSL.
[00:45.870 --> 00:52.010]  The protocol, a number of implementations, all have gotten beaten up pretty badly, even in the mainstream press.
[00:53.230 --> 00:57.670]  So a few of these attacks or vulnerabilities might ring a bell to you.
[00:57.670 --> 01:04.630]  There was Crime, Breach, Beast, Lucky 13, Heartbleed, Three Shake.
[01:04.810 --> 01:09.010]  Basically, you couldn't have an SSL vulnerability without a nifty little name.
[01:09.770 --> 01:12.770]  And making basically like headline news with almost all of these.
[01:13.050 --> 01:16.010]  So, man, bad times all around.
[01:16.330 --> 01:18.990]  But there's something else out there, way more prevalent.
[01:18.990 --> 01:24.470]  It's been around for a longer time, not quite as sexy a name, but get ready for this big animated reveal.
[01:25.130 --> 01:27.770]  Yell it out when you think you know what I'm going to talk about here.
[01:28.230 --> 01:28.330]  Please.
[01:28.570 --> 01:29.130]  Not HTTP.
[01:29.930 --> 01:30.730]  Exactly, right?
[01:31.530 --> 01:31.970]  HTTP.
[01:32.450 --> 01:32.710]  Look at it.
[01:33.050 --> 01:33.870]  There it is.
[01:33.990 --> 01:34.370]  There it is.
[01:34.430 --> 01:34.830]  HTTP.
[01:35.510 --> 01:36.810]  Hypertext Transport Protocol.
[01:37.190 --> 01:38.310]  It's all over the Internet.
[01:38.550 --> 01:40.150]  And it's a big problem today.
[01:43.970 --> 01:52.250]  So, one thing I love about hacker conferences is that you get people from all walks of life, all different levels of technical ability in one room to learn about some stuff.
[01:52.490 --> 01:53.770]  So, this is something for the noobs.
[01:54.450 --> 01:56.370]  If you're super leet, you know this already.
[01:56.610 --> 01:59.390]  But let me break down for you why HTTP is a bad thing.
[01:59.850 --> 02:00.730]  Here you are.
[02:01.190 --> 02:03.870]  You're at HOPE or you're at a coffee shop downstairs.
[02:04.390 --> 02:06.170]  And you want to browse the Internet, right?
[02:07.150 --> 02:20.590]  So, you're connecting via some free, unencrypted public Wi-Fi to some machine that is probably hosted in the back room or under the espresso machine of this coffee shop.
[02:21.890 --> 02:24.990]  It then connects out over the Internet as we know it.
[02:25.150 --> 02:26.510]  So, there are a whole bunch of different hops.
[02:26.630 --> 02:32.730]  And who really knows what happens in between the coffee shop and was ultimately us, your destination.
[02:32.970 --> 02:34.210]  So, in this case, you can call it Twitter.
[02:35.770 --> 02:39.850]  As we all know, there are a lot of these folks probably here today.
[02:39.850 --> 02:42.570]  I think I saw a guy wearing this hat in the elevator earlier.
[02:44.270 --> 02:49.730]  But, there are people out there that might be interested in looking at what you're doing online.
[02:49.990 --> 02:53.290]  So, surveillance is a big topic of this conference.
[02:53.870 --> 02:55.710]  And also, maybe mucking around with it.
[02:55.770 --> 02:56.950]  Having a good time at your expense.
[02:57.690 --> 03:07.410]  The key thing to note is that every hop in this node presents an opportunity for somebody to sniff your traffic, manipulate your traffic, or do something else nasty.
[03:09.170 --> 03:10.570]  Man, this is super funky.
[03:11.330 --> 03:14.310]  But, don't just take my word for it, right?
[03:14.490 --> 03:17.810]  So, this is a screenshot that has been terribly distorted.
[03:18.170 --> 03:19.730]  It looks fine on my machine.
[03:19.950 --> 03:21.510]  But, of Wireshark, right?
[03:22.370 --> 03:29.150]  And, what I'm showing here, like this is me in a coffee shop, capturing traffic with the promiscuous network card.
[03:30.830 --> 03:34.250]  And, looking at the ones and zeros as they fly through the air.
[03:34.390 --> 03:36.470]  Assembling them into TCP and then HTTP.
[03:36.930 --> 03:40.490]  And then being able to kind of like click in and jump into what's actually being sent on the wire.
[03:41.710 --> 03:45.590]  So, again, this is all super fuzz and mutated because of the display.
[03:45.930 --> 03:49.870]  But, what you'll see here if you dig in is that, hmm, this is interesting.
[03:49.870 --> 03:53.470]  It's an HTTP post to a login endpoint with a username.
[03:53.910 --> 03:54.930]  Say, my username.
[03:55.210 --> 03:57.170]  And, a password that I just put, yeah, right.
[03:57.390 --> 03:58.550]  It's not my actual password.
[03:58.810 --> 04:00.630]  But, it's flying over the network in clear text.
[04:00.950 --> 04:04.830]  So, we've been logging in over SSL for a long time now.
[04:04.910 --> 04:09.250]  A lot of sites, even the worst sites on the Internet, don't accept your username and password over HTTP.
[04:10.590 --> 04:15.690]  It was interesting to see the EFF talk not too long ago where there were some sites that still take username and password over HTTP.
[04:15.690 --> 04:19.750]  But, for the most part, we kind of got that solved.
[04:19.890 --> 04:24.630]  We're using HTTPS for username and passwords because a lot of people reuse passwords on different sites.
[04:25.010 --> 04:26.370]  So, that's good.
[04:27.070 --> 04:36.050]  What is not good is if you log in over HTTPS and then you downgrade people to HTTP, these session cookies are flying around the network in clear text.
[04:36.210 --> 04:45.390]  And so, you might not steal anybody's username and password, but you'll be able to sniff out, steal, and playback authentication cookies that are used for logging into a site.
[04:45.390 --> 04:50.790]  So, as you probably know, when you log into twitter.com, you type your username and password once.
[04:51.090 --> 04:52.110]  We give you a cookie.
[04:52.290 --> 04:56.970]  Your browser holds onto it and sends it up to us every single time you make a request so that we know who you are.
[04:58.190 --> 05:02.970]  In the world of plain text, Internet traffic, things like this come about.
[05:03.330 --> 05:04.830]  So, this is an old screenshot.
[05:04.990 --> 05:05.290]  2010.
[05:05.510 --> 05:06.690]  This is super old Twitter.
[05:07.050 --> 05:11.750]  I think for the sake of everybody's eyeballs, I'm going to unplug this and plug it back in.
[05:23.260 --> 05:24.120]  And then pray.
[05:30.540 --> 05:31.180]  I don't know.
[05:31.320 --> 05:31.760]  Alas.
[05:34.040 --> 05:35.660]  Is that even worse than it was before?
[05:35.820 --> 05:36.320]  I'm not sure.
[05:36.640 --> 05:38.100]  But anyways, this is Firesheep.
[05:38.340 --> 05:40.480]  Again, super funked out Firesheep.
[05:41.060 --> 05:47.680]  But it's an HTTP sniffing extension that you can add to your Firefox browser.
[05:47.680 --> 05:52.440]  It will allow you to, again, smell the ones and zeros as they fly through the air.
[05:53.160 --> 05:58.680]  See who is logging into different websites with HTTP supporting traffic.
[05:59.420 --> 06:01.220]  So, this is back in the day, like I said, 2010.
[06:01.820 --> 06:04.220]  If you could see here, there's Twitter.
[06:04.520 --> 06:05.140]  There's Facebook.
[06:05.420 --> 06:06.140]  There's Yahoo Mail.
[06:06.360 --> 06:07.100]  I think there was Hotmail.
[06:07.240 --> 06:07.780]  There's Amazon.
[06:08.200 --> 06:15.280]  And you, as a kind of nasty hacker bad guy, can just select whichever person that you want to log in as.
[06:15.500 --> 06:16.840]  You don't need to know their username.
[06:17.000 --> 06:17.740]  You don't need to know their password.
[06:17.840 --> 06:18.380]  You just look around.
[06:18.860 --> 06:20.820]  You say, that guy looks like an interesting guy to log in as.
[06:20.900 --> 06:23.380]  And you click his name and you're signed in to whatever service you want as him.
[06:24.500 --> 06:25.560]  So, again, that was 2010.
[06:25.880 --> 06:26.600]  That was back in the day.
[06:26.740 --> 06:29.120]  This is sort of pre-Snowden revelation.
[06:30.060 --> 06:34.740]  But what was nice about FireSheep is that it shone a light on an issue that existed forever.
[06:34.960 --> 06:36.760]  Like, you don't need FireSheep to do this.
[06:36.840 --> 06:38.800]  You can just use Wireshark and write a script yourself.
[06:40.340 --> 06:44.360]  But FireSheep was kind of the first mainstream adoption and realization that this is a problem.
[06:44.480 --> 06:48.240]  And it's really easy to man-in-the-middle HTTP traffic.
[06:48.540 --> 06:53.220]  So, as a result, at Twitter, that was like December 2010 that FireSheep came out.
[06:53.620 --> 06:58.800]  January 2011, we rolled out an option to use HTTPS all the time on your Twitter.com sessions.
[06:58.800 --> 07:03.760]  So, what we had here is like you would dig into your settings page.
[07:04.020 --> 07:04.780]  We announced it.
[07:04.880 --> 07:08.720]  But in order to use this feature, you had to go into settings and click it.
[07:09.280 --> 07:16.640]  And then from that point forward, you'd be safe while you were logged in from FireSheep type attacks, right?
[07:16.740 --> 07:20.620]  Like, once you were authenticated, it was SSL from that point forward and nothing else.
[07:20.980 --> 07:21.940]  That was great.
[07:22.120 --> 07:23.940]  People used it.
[07:24.100 --> 07:26.420]  We ironed out a few bugs in the process.
[07:26.420 --> 07:33.520]  But what kind of sucks is that, like, you only get the people that dig deeply down into their settings and really care about security.
[07:33.840 --> 07:36.760]  Ideally, we'd be able to protect everybody all the time.
[07:36.940 --> 07:39.260]  And you'd have to go out of your way to do something insecure.
[07:39.500 --> 07:48.740]  So, over the course of a few months, we gradually turned up a little knob to make this default behavior is opt-out for HTTPS all the time.
[07:48.740 --> 07:55.360]  So, you log in now, and unless you go in and uncheck the box, you get SSL, kind of whether you like it or not.
[07:57.300 --> 08:04.920]  After cranking that up to 100% and seeing only a few people opt-out, we just took away the option altogether.
[08:04.920 --> 08:09.800]  So, we said, alright, if you want to use twitter.com, SSL, tough luck, you're going to get it in 2012.
[08:12.000 --> 08:19.600]  So, I mentioned that, you know, surveillance is something that we all care deeply about here, as is sort of censorship.
[08:19.900 --> 08:25.840]  And so, I'm going to sprinkle in a few, like, wins and good stories that we saw as a result.
[08:28.000 --> 08:28.440]  Alright.
[08:30.520 --> 08:32.020]  That looks better, at least.
[08:32.780 --> 08:34.120]  And so, that's tough to read.
[08:34.220 --> 08:45.200]  And what I want to highlight down here is that there was a situation in India, some years ago, where a little, you know, political discontent stuff going on.
[08:45.420 --> 08:49.420]  And the government wanted to censor certain twitter accounts from being accessed.
[08:49.700 --> 08:51.780]  And they did so in a pretty naive way.
[08:51.780 --> 08:56.300]  They gave all the ISPs in the country URLs to say, block these URLs, basically.
[08:56.620 --> 09:00.620]  Don't let anybody visit twitter.com slash activist1.
[09:02.020 --> 09:17.160]  What's great about HTTPS, and this has been mentioned before, too, is that after the handshake and you make your SSL connection, everything in the URL, so whether it be the path or the query string, is totally blind, or anybody in the middle is totally blind to what that could be.
[09:17.160 --> 09:31.900]  So, what people realized is that if they went to a profile for somebody that had been blocked by the government, all they had to do was actually, like, sign in, and then they would access the site over HTTPS, and then, like, hurrah, you get to see the stuff that the government had tried to block.
[09:32.080 --> 09:33.340]  So, that was a win.
[09:33.480 --> 09:34.300]  It was really easy.
[09:34.620 --> 09:36.440]  And it made the government look kind of dumb.
[09:36.480 --> 09:37.120]  So, it's even better.
[09:40.020 --> 09:47.940]  So, one thing to note, and if I don't say this, somebody's going to yell at me from the crowd and say it for me, so I'll get in front of it and mention this now.
[09:48.360 --> 09:57.540]  Even if you have sessions that are 100% SSL all the time, this guy still has a few tricks that he could use to try to steal your cookies.
[09:58.540 --> 10:03.060]  One gift he might have for you is a HTTP embed on one of his pages.
[10:03.260 --> 10:04.180]  It's not on Twitter.com.
[10:04.280 --> 10:07.180]  He can send you an email that can contain this HTML tag.
[10:07.460 --> 10:17.560]  And the way browsers work, again, is that once you get a cookie from a domain, if you don't do anything special with it, it's going to send that cookie to the domain every time you make a request there.
[10:17.780 --> 10:25.680]  So, again, if you're signed into Twitter.com, you're going to make this fetch, and you're going to send your cookie along, oh no, it's over HTTP.
[10:26.040 --> 10:27.720]  He'd be able to steal it and send it back to us.
[10:28.500 --> 10:30.460]  Unless, of course, you mark your cookie secure.
[10:30.740 --> 10:33.440]  So, this is, again, kind of old news.
[10:34.600 --> 10:37.260]  But if I didn't say it, somebody would call out and yell at me.
[10:37.400 --> 10:39.380]  So, if only security were always this easy.
[10:39.500 --> 10:41.640]  We would just pass down an option that says, yeah, make it secure.
[10:42.340 --> 10:44.280]  But for a set cookie, that's really the case.
[10:44.420 --> 10:47.440]  Mark it secure, and it won't fly out over the network in clear text.
[10:49.460 --> 10:54.560]  So, it was a very gradual process towards getting everything on SSL all the time.
[10:54.780 --> 10:58.260]  For a while, we'd allow you to load HTTP Twitter.com.
[10:58.360 --> 11:01.000]  Like I said, all of our authenticated sessions were SSL.
[11:01.240 --> 11:05.160]  Our unauthenticated sessions, we still let you browse a site not logged in over HTTP.
[11:05.840 --> 11:11.400]  So, what that means is that you load HTTP Twitter.com, and you end up with a page like this.
[11:11.660 --> 11:13.940]  This was mentioned, again, in some of the earlier talks.
[11:13.940 --> 11:20.140]  But how do you know where this post is going to end up when you click sign in, right?
[11:20.220 --> 11:28.060]  Like you put your username and password in there, but there's nothing indicating to you that this is going to go over a secure channel, or to some other website that Twitter doesn't even own.
[11:28.600 --> 11:34.040]  So, there's an attack that Moxie Marlinspike popularized, along with SSLSniff.
[11:34.180 --> 11:44.320]  There's SSLstrip, which basically takes the page that has been served up over HTTP, and substitutes HTTP links for every HTTPS link that it sees.
[11:44.600 --> 11:51.440]  So, in this case, I'll kind of go to exhibit A, which was his first screenshot, and exhibit B, which is his second screenshot.
[11:51.760 --> 11:53.960]  One of these has been SSLstrip, the other hasn't.
[11:54.560 --> 11:56.100]  And they look exactly the same, right?
[11:56.280 --> 12:09.680]  So, what the point of this slide is, is that even though we moved everybody over to SSL once they logged in, there was still this window of opportunity for attackers to get you before we get you into a safe space with SSL.
[12:10.020 --> 12:11.840]  So, what do we do about that?
[12:13.700 --> 12:16.040]  This is a bit of a motivational image.
[12:16.200 --> 12:24.780]  Again, it would be fantastic if this thing would actually look good when I showed it to you, but to kind of represent the next level shit that we've done that I'm going to talk about here.
[12:26.480 --> 12:30.340]  And, you know, most good tech talks have a disclaimer of some sort.
[12:30.940 --> 12:39.480]  You know, I'm not a lawyer, and the disclaimer here is that we did all this stuff, and I'm up here because you can too, and I would like you to do this.
[12:41.100 --> 12:46.940]  So, let's take a step back a little bit, a little more handcrafted slide work from me.
[12:47.760 --> 12:50.780]  But, say that you are just a Joe user.
[12:50.920 --> 12:52.020]  You hear about this Twitter thing.
[12:52.020 --> 12:57.360]  Ellen tweeted some funny selfie, or Geraldo is taking pictures of himself in the bathroom mirror.
[12:57.740 --> 12:59.620]  You want to find out what Twitter is all about.
[12:59.880 --> 13:04.240]  So, you go into your browser, and you open your location bar, and you just type Twitter.
[13:05.420 --> 13:10.980]  This is kind of a standard interview question for some, like, networking jobs and stuff like that.
[13:11.120 --> 13:14.260]  So, you know, what happens when you type Twitter into the location bar?
[13:15.380 --> 13:25.260]  In this case, the user didn't qualify with, like, a .com or a .net, so actually the default behavior you're going to get is you're going to end up on some browser, or some search engine, right?
[13:25.740 --> 13:27.160]  So, maybe you use Bing.
[13:28.000 --> 13:29.140]  Maybe you use Yahoo.
[13:30.000 --> 13:31.220]  Maybe you use Google.
[13:31.540 --> 13:34.680]  And so, I'll go back a few slides here.
[13:35.080 --> 13:36.760]  One of these things is not like the other.
[13:37.040 --> 13:39.180]  So, first result for Bing, Twitter.
[13:40.100 --> 13:41.860]  Yahoo, Twitter.com.
[13:43.240 --> 13:46.140]  But Google's URL is HTTPS, Twitter.com.
[13:46.140 --> 13:51.280]  Again, this is before we moved everything over to SSL all the time, logged in or logged out.
[13:51.560 --> 13:53.420]  So, what's this crazy magic?
[13:53.560 --> 13:54.740]  How do we pull this one off?
[13:57.040 --> 14:00.680]  You might not be surprised to know that there's a lot of name calling on the Internet.
[14:01.000 --> 14:04.820]  And this is, like, bullies and meanies, or it's DNS and CNames.
[14:05.160 --> 14:16.800]  So, we serve up content when you make a request to our servers, and it's possible for people to actually put their domain names in front of our servers, and we can't really do too much about it.
[14:16.940 --> 14:32.120]  So, we can detect it when it happens, but there's nothing really stopping you from saying, I'm going to register, you know, HopeTwitter.com, and I'm going to point it over here, so that when you make a request to HopeTwitter.com, it gets looked up and ends up getting content served from our servers.
[14:33.720 --> 14:37.420]  And so, this does all sorts of bad things for, like, the business and for the website.
[14:37.420 --> 14:40.680]  It lets your content get indexed on other domains.
[14:40.740 --> 14:44.780]  It makes people feel comfortable potentially logging into other domains that you don't own.
[14:44.920 --> 14:47.720]  And it's generally a thing that you don't want to have happen.
[14:47.980 --> 15:06.340]  So, Google supports this link rel canonical element that you can embed in the HTML on your page, and it lets you, as a server that's rendering the content, tell a crawler or a search engine bot where this content is authoritatively being served from.
[15:06.340 --> 15:10.680]  So, again, we had HTTP Twitter.com links out there.
[15:10.840 --> 15:15.280]  We had HTTP Twitter.wacky domain stuff out there.
[15:15.380 --> 15:24.760]  But we always served up the content with this hidden tag in there that told the crawlers that, no matter how you got to this site, index it as if it came from HTTPS Twitter.com.
[15:24.980 --> 15:33.740]  So, as a result of that, we saw the HTTPS URL kind of jump up the chain in the search engine results.
[15:33.740 --> 15:41.940]  And what's nice, the big defense against SSL strip is never letting people try to access your site over HTTP in the first place, right?
[15:42.080 --> 15:52.780]  So, if you do an encrypted web search, you're on HTTPS Google.com, and then the link that you click is to HTTPS Twitter.com, no HTTP request is happening there, no SSL strip, no SSL sniff.
[15:53.600 --> 15:54.060]  Great win.
[15:54.060 --> 15:55.300]  So, that was fantastic.
[15:55.980 --> 16:01.600]  I mentioned this, like, again, this is kind of old news for us, but not a lot of people do it, so that's why I want to tell people about it.
[16:01.760 --> 16:07.060]  It's one line of code, and you can index your content with a secure link.
[16:08.100 --> 16:15.040]  There was a blog post that somebody pointed out to me from some of these, like, search engine optimization folks that said this is actually a bad thing to do.
[16:15.120 --> 16:24.320]  You're going to confuse the crawlers, and you're going to end up with negative results when it comes to your URL getting indexed.
[16:24.340 --> 16:32.380]  And so, I think that we saw this maybe that it took a little while for the HTTPS URL to creep up to the top, but it's not like we lost credibility in the process.
[16:32.620 --> 16:39.380]  And so, one little funny aside, again, is that any web hackers in the audience know what this URL encoded value is?
[16:40.420 --> 16:40.940]  That's right.
[16:41.140 --> 16:47.920]  So, I was doing something where I was, you know, maybe some directory traversal attack where I was like, yeah, what was the URL encoding for slash again?
[16:48.080 --> 16:52.420]  So, you hit up Google, and you just do Q equals percent 2F, because you're like, ah, let me check this out.
[16:52.940 --> 17:00.960]  Funny enough, the number one search result for slash is, like, slash on Twitter, and it's an HTTPS site.
[17:01.080 --> 17:05.200]  So, rock and roll, everybody wins, secure, and all the better.
[17:06.640 --> 17:10.680]  So, Bing and Yahoo, these dudes are still loading the page over HTTP.
[17:11.100 --> 17:11.480]  What gives?
[17:12.160 --> 17:25.900]  They say that in order to be indexed over HTTPS, you need to respond with a 301, which is a HTTP response code telling the crawler or anybody that visits this site, no, we permanently moved this resource to this HTTPS URL over here.
[17:26.560 --> 17:32.420]  So, we were like, okay, 301 redirects are coming up, freshly served from our front end to you, the user.
[17:32.980 --> 17:36.040]  Any time you load HTTP, twitter.com, you're going over to HTTPS.
[17:37.840 --> 17:40.080]  It took a little while for them to catch up.
[17:41.180 --> 17:43.600]  Like, literally a pretty long time.
[17:44.800 --> 17:47.780]  Our search results are still HTTP, twitter.com, on Bing.
[17:48.120 --> 17:49.760]  Still HTTP, twitter.com, on Yahoo.
[17:49.900 --> 17:51.940]  Or, HTTP, twitter.com, on Yahoo.
[17:52.640 --> 17:53.540]  What's the deal here?
[17:53.740 --> 18:01.220]  So, eventually, after some time, I'd given versions of this exact talk at smaller, like, B-sized conferences in Seattle and San Francisco.
[18:01.740 --> 18:06.220]  And before I updated my slides for hope, I was like, let me just go back.
[18:06.380 --> 18:09.320]  Hopefully, by now, we're indexed as HTTPS somewhere.
[18:09.560 --> 18:11.420]  So, Bing caught up.
[18:11.520 --> 18:15.360]  Now, we're HTTPS, twitter.com, on the front page of Bing when you look.
[18:15.860 --> 18:16.060]  All right.
[18:16.240 --> 18:16.780]  Almost there.
[18:17.340 --> 18:18.980]  Yahoo was still HTTP.
[18:19.420 --> 18:20.140]  So, this was a bummer.
[18:21.240 --> 18:29.100]  What I did, actually, recently, while I was preparing for this talk, is email the chief security officer of Yahoo and was like, this isn't a threat.
[18:29.360 --> 18:31.320]  I'm about to give this talk in a few weeks.
[18:31.440 --> 18:35.360]  And I've noticed that we've been serving up this 301 redirect for, like, a year plus now.
[18:35.520 --> 18:36.780]  And we're still indexed as HTTP.
[18:37.600 --> 18:40.520]  As a result, now we're indexed as HTTPS, twitter.com.
[18:40.640 --> 18:48.720]  And so, that was an effective way to get us basically linked as the number one search result for Twitter with HTTPS URLs.
[18:48.920 --> 18:49.500]  So, that's nice.
[18:49.600 --> 18:57.660]  If you're trying to find Twitter through a search engine because you don't know that you should probably just do this, then you're protected.
[18:58.120 --> 18:59.760]  Which brings me to this next point.
[18:59.840 --> 19:13.080]  If you're a little more tech savvy and you type twitter.com into your browser because you think that's where Twitter might live, are you actually less secure than you might be if you were coming to us via a search engine?
[19:13.680 --> 19:15.180]  And you would think, yes, right?
[19:15.280 --> 19:19.320]  Like, by default, the browser behavior is to try HTTP first.
[19:19.480 --> 19:23.540]  So, you type twitter.com, it's going to make an outbound HTTP twitter.com request.
[19:23.720 --> 19:26.720]  It will get the 301 and then it will send you to HTTPS.
[19:26.920 --> 19:35.240]  But that gives the attackers that are man in the middling us an opportunity to basically never send us to SSL version of Twitter at all.
[19:36.080 --> 19:41.260]  And to basically perform SSL strip attacks and steal our cookies and steal our passwords and all that stuff.
[19:41.640 --> 19:45.960]  So, I wouldn't be telling you about this if we didn't do something to address this issue as well.
[19:47.020 --> 19:53.660]  And so, an opportunity for you to play along at home, if you want, is type twitter.com into Chrome or Firefox.
[19:54.920 --> 19:56.180]  And you'll notice this behavior.
[19:56.320 --> 20:02.660]  So, open up Wireshark if you want, or open up the network tab in the developer tools, and you'll see that you get a redirect.
[20:03.640 --> 20:05.380]  Oh my God, it looks terrible up there.
[20:05.780 --> 20:11.640]  But, you'll get a redirect that happens locally before ever hitting the network.
[20:11.940 --> 20:22.920]  And so, if you could see this, there's this arrow pointing down here that says twitter.com, pending, and then a redirect over there to HTTPS twitter.com.
[20:23.020 --> 20:29.720]  So, the browser did try to load the resource over HTTP, but it itself knew, no, no, no, no, no.
[20:29.740 --> 20:32.380]  We're going to go SSL before we actually make this outbound request.
[20:33.300 --> 20:34.880]  And what is this all about?
[20:35.120 --> 20:49.940]  The HSCS header has been mentioned a few times today, but HTTP strict transport security is a header that you can set down when you visit a site over SSL that tells the browser, kind of like, oh, phew, you made it here over SSL.
[20:50.000 --> 20:53.960]  Let's never go back to that scary place of plain text HTTP traffic ever again.
[20:54.160 --> 21:08.840]  So, from this point forward, until this max age expires, which is basically 20 years in this case, if you ever get an HTTP twitter.com reference, turn it to HTTPS locally on the client before you do anything else.
[21:09.720 --> 21:11.100]  So, this is fantastic.
[21:11.400 --> 21:14.240]  It actually defeats SSL strip, basically.
[21:15.400 --> 21:16.580]  And so, it's funny.
[21:16.780 --> 21:19.520]  You know, Moxie, the author of SSL strip, he worked at Twitter for a little while.
[21:19.780 --> 21:20.700]  Super legit dude.
[21:20.900 --> 21:21.460]  He was my boss.
[21:21.580 --> 21:22.340]  He's a really cool guy.
[21:22.920 --> 21:32.780]  But it's not as if we got defense against SSL strip by hiring the author and then telling him to make it not work against Twitter anymore, which some people thought we did.
[21:33.400 --> 21:35.140]  We actually just use the HSTS header.
[21:35.320 --> 21:40.820]  So, you, too, can protect against SSL strip without hiring the author and just send down this header.
[21:41.000 --> 21:43.380]  It's one line in your Apache config, whatever, and you're all set.
[21:45.720 --> 21:51.840]  Even better than HSTS is the ability to preload HSTS into the browsers themselves.
[21:51.840 --> 21:59.440]  So, let's say that you've never been to Twitter.com and you never got redirected to SSL and you never got the HSTS header.
[22:00.100 --> 22:06.140]  It's basically physically impossible unless you modify the binary to load HTTP Twitter.com in Chrome.
[22:06.640 --> 22:12.660]  And that's because, similar to, you know, emailing the CSO at Yahoo and being like, hey, dude, let's get the SSL over here.
[22:12.920 --> 22:14.480]  You can email Adam Langley.
[22:14.780 --> 22:16.180]  He's probably here somewhere, maybe.
[22:16.980 --> 22:21.700]  Or some developers on the Chrome team and kind of pre-cook HSTS into the browser.
[22:21.700 --> 22:23.860]  So, Chrome started doing this first.
[22:24.840 --> 22:27.260]  Firefox has picked up their list and started doing it as well.
[22:27.520 --> 22:33.700]  All you have to do is tell them that, yeah, we don't want people to ever visit HTTP resources using your browser ever.
[22:33.960 --> 22:42.880]  So, if you load Twitter.com, like I said, in a fresh install that's never been on the network before, you type in HTTP Twitter.com, it's going out over SSL.
[22:42.980 --> 22:45.240]  It's getting redirected locally before it hits the network.
[22:45.420 --> 22:47.900]  Again, no SSL strip, cooked into the clients.
[22:48.520 --> 22:49.460]  This is awesome.
[22:51.840 --> 22:54.160]  So, we rolled all this stuff out gradually.
[22:55.640 --> 23:01.680]  And measured some things, but didn't really, like, keep a super close ID on it and just, like, flip the switch all at once.
[23:01.820 --> 23:08.660]  We were pretty gradual in rolling out both opt-out SSL and basically removing the ability to opt-out SSL.
[23:08.660 --> 23:09.940]  All these things happened kind of gradually.
[23:10.900 --> 23:19.940]  We had the luxury then, after we knew that it worked to flip the switch on some of our subdomains that are not on the Twitter.com domain and then monitor what happened.
[23:20.200 --> 23:24.000]  So, this particular graph shows kind of the trend.
[23:24.280 --> 23:26.420]  It looks like somebody's dying here or something.
[23:26.660 --> 23:35.460]  But, what we're counting is the number of HTTP to HTTPS redirects we're doing when you load web.tweetdeck.com.
[23:35.660 --> 23:41.160]  So, what would happen is you would type in, you know, I want to use the tweetdeck web client.
[23:42.200 --> 23:46.300]  And we would say, oh, you hit us over HTTP, you need to go to SSL instead.
[23:46.740 --> 23:56.060]  What's apparent here is when we started setting the HSTS header because the number of HTTP requests we got to this domain, like, dropped off almost entirely.
[23:56.320 --> 24:04.020]  So, granted, like, about a third of the traffic, maybe still, came to us over HTTP.
[24:04.300 --> 24:09.800]  And that's because this happened back in the day when not every browser supported HSTS.
[24:09.920 --> 24:13.500]  I think IE still doesn't, but is going to do it soon.
[24:15.260 --> 24:17.400]  But, so, this is not like people stopped using the site.
[24:17.500 --> 24:22.880]  They just stopped using the site in a way that would make them susceptible to SSL strip attacks.
[24:23.180 --> 24:24.720]  So, this is great to see in practice.
[24:24.900 --> 24:25.840]  We were like, yeah, it worked.
[24:25.980 --> 24:26.280]  Awesome.
[24:27.060 --> 24:31.800]  And kind of gave us this mantra that we abide by inside Twitter.
[24:31.960 --> 24:40.400]  So, if you join Twitter engineering, on your third day, you hear me or somebody from my team give a talk about writing secure code at Twitter and all the things we do.
[24:40.600 --> 24:43.400]  And this is a slide we pull from, like, the new hire orientation deck.
[24:43.560 --> 24:45.720]  Like, we jam it down your throat on day one.
[24:46.140 --> 24:50.140]  And it's, like, a fantastic law to abide by.
[24:50.260 --> 24:52.660]  And, again, I'm here to tell more people to do the same thing.
[24:53.480 --> 24:56.060]  There's an earlier kind of jab.
[24:56.200 --> 24:56.920]  It's like, come on, everybody.
[24:57.040 --> 24:57.480]  It's 2014.
[24:57.560 --> 24:59.340]  We can use HTTPS.
[25:00.380 --> 25:09.460]  But when you do, once you start living by this mantra, it's still possible for you to make mistakes or other people to kind of make mistakes on your behalf and affect the integrity of your SSL implementation.
[25:11.200 --> 25:21.740]  So, this little dude, the yellow lock, the broken lock, is an indicator that somebody's done something wrong with your SSL content that's been loaded up.
[25:21.800 --> 25:22.820]  And it's probably the developers.
[25:23.120 --> 25:27.740]  And so, what we use as a little trick is content security policy.
[25:27.740 --> 25:30.800]  So, this is something that I'm not aware of many other people doing.
[25:31.060 --> 25:33.840]  And, again, I hope more people take from this talk and start doing.
[25:34.440 --> 25:38.880]  Get ready for a super busy slide that is going to be super tough to read.
[25:39.260 --> 25:43.900]  But this is a CSP header that we send down with some of the content that you load.
[25:44.000 --> 25:46.460]  And I'll walk through it a little bit to tell you what's happening here.
[25:46.860 --> 25:52.560]  So, similar to how HSTS tells the browser that, great, you connected over HTTPS.
[25:52.840 --> 25:54.400]  Let's never go back to HTTP again.
[25:55.320 --> 25:58.900]  CSP tells the browser, okay, fine, here's all this resources that you load.
[25:59.260 --> 26:02.320]  Now, here's some stuff not to do with the code that we just gave you.
[26:03.320 --> 26:06.940]  So, it starts off, you know, the HTTP header, content security policy.
[26:08.180 --> 26:17.000]  After that, we use a whole bunch of different directives, but you can tell what we're trying to accomplish here is, like, if you want to load some content, it needs to start with the HTTPS protocol.
[26:18.100 --> 26:21.700]  The iframes that we load on our site, okay, they need to start with HTTPS, whatever.
[26:21.700 --> 26:24.900]  We're not enforcing that all the content needs to come from twitter.com.
[26:25.040 --> 26:32.200]  You can see there's a couple examples here where, in order to be backwards compatible with some stuff, we allow a few HTTP connections.
[26:33.140 --> 26:43.440]  But what is nice about this is that if you have it in just default mode, which is content security policy, the browser is actually going to block any case where we messed up.
[26:43.600 --> 26:52.820]  Like, it's going to say, oh, it looks like the developers include an HTTP link or HTTP, not a link, but HTTP resource in this page, I'm just not going to load it.
[26:53.100 --> 26:57.460]  And what's great is it, like, will notice it as developers, because things break when we do the wrong thing.
[26:58.240 --> 27:03.140]  Even better than that is that CSP has a support for our report URI directive.
[27:03.420 --> 27:14.300]  And so when one of these violations happen, the browser is actually going to make a post back to a URL that we are going to be specifying the header and it's going to tell us, like, we did something wrong.
[27:14.480 --> 27:19.360]  When we tried to load this page and something that we told it to not do was done.
[27:19.540 --> 27:25.980]  And so we have some basically, like, you know, real-time aggregation stuff going on.
[27:26.080 --> 27:30.000]  And we can basically tell if we shipped a bug to production, oh, no, this thing slipped out.
[27:30.120 --> 27:31.540]  And then we have the HTTP resource.
[27:32.660 --> 27:36.440]  Let's fix that up in the next deploy and get everything back to the nice green lock.
[27:36.660 --> 27:39.260]  So use CSP for a number of things.
[27:39.420 --> 27:51.180]  It's traditionally thought of as, like, cross-site scripting mitigation, but you can use it for getting and detecting any usage of HTTP on your HTTPS site.
[27:51.300 --> 27:52.300]  So it's great.
[27:53.500 --> 27:59.900]  It's also tough to apply uniformly across all your different properties in the same way all the time.
[28:00.360 --> 28:02.520]  And we have HSTS.
[28:02.620 --> 28:05.680]  We have content security policy.
[28:05.800 --> 28:11.800]  There are all these other headers that you should send down as well to tell the browser not to mess up this stuff that you're trying to display.
[28:12.160 --> 28:15.440]  And so we created a library that we've open-sourced.
[28:15.540 --> 28:16.700]  It's up on GitHub.
[28:17.320 --> 28:19.480]  It basically does the right thing by default.
[28:19.720 --> 28:30.980]  So I mentioned earlier with the SSL all the time on twitter.com how it was crucial for us to make the right thing happen by default and not rely on our users going in and checking the checkbox.
[28:31.200 --> 28:32.920]  Same thing with our developers at Twitter.
[28:33.500 --> 28:35.600]  We want them to use this library.
[28:35.600 --> 28:39.020]  It's been ported to like Go and all these different languages.
[28:39.360 --> 28:44.360]  Like if you visit this site we have links to the other languages it's been ported to.
[28:44.580 --> 28:48.400]  But for developers too we just want you to do the right thing without having to think about it.
[28:48.540 --> 28:49.920]  So this library is out there.
[28:50.080 --> 28:50.780]  Please contribute to it.
[28:50.880 --> 28:51.420]  Please use it.
[28:51.960 --> 28:52.600]  Check it out.
[28:52.840 --> 28:54.960]  It will do all this stuff for you without thinking about it.
[28:55.620 --> 29:01.200]  Another good way to detect when you might make a mistake is just to make things break hard when you do.
[29:01.200 --> 29:09.700]  So after we moved the whole http twitter.com site to https we still had the platform to worry about, right?
[29:09.860 --> 29:14.200]  Like our clients connect to us over api.twitter.com.
[29:14.340 --> 29:27.880]  A lot of our third-party partners connect to us over api.twitter.com which was sort of a longer tale to fix because we have partners that like we can't just say okay boom https now for everybody and get ready for it.
[29:28.620 --> 29:37.960]  So what we did there is kind of gradually tell people about it, tell the developers that we're going to move api.twitter.com to SSL all the time, get ready for it.
[29:38.520 --> 29:48.480]  And we ran a series of like blackout tests you might call it where the way you might notice that we've made this change is that your app just breaks and it stops working.
[29:48.740 --> 29:54.880]  And so in order to be kind of nice to people we say okay we're going to turn off http for an hour on this day.
[29:55.340 --> 29:57.620]  Everything's going to break if you're not over on SSL yet.
[29:58.420 --> 30:03.520]  And people that aren't paying attention to like our mailing list and stuff are going to be like oh what the heck happened my thing stopped working.
[30:03.760 --> 30:12.560]  And then they're going to yell at us about it and we're going to say yeah we were running this test so just add an S to your httpapi.twitter.com URL and then you'll be all set.
[30:13.160 --> 30:22.360]  So we did this over the course of like a few months and then now the api.twitter.com endpoint will basically return this error if you try to use http.
[30:22.800 --> 30:26.720]  So this is great because not everybody uses http.twitter.com.
[30:26.820 --> 30:28.140]  Like I said we have third-party clients.
[30:28.300 --> 30:29.800]  We've got our own native clients.
[30:29.980 --> 30:33.780]  But everybody there now is using SSL all the time because the server enforces it.
[30:34.700 --> 30:35.800]  So this is great.
[30:36.560 --> 30:37.860]  There's all the ways you can mess up.
[30:37.960 --> 30:42.960]  There are a number of documented situations where somebody else messes up SSL for you.
[30:43.160 --> 30:50.300]  So there is the TLS protocol itself and there's a dependency on the certificate authorities.
[30:50.880 --> 30:52.680]  And trust needs to come from somewhere.
[30:53.140 --> 30:55.280]  So there's this paper.
[30:55.500 --> 31:04.200]  There's this incriminating slide that names a number of folks that have been involved in one way or another in subverting the trust model that TLS depends on.
[31:04.600 --> 31:10.840]  But basically you need to get the trust that you're talking to, who you think you're talking to from somewhere.
[31:11.020 --> 31:12.920]  And that's where certificate pinning comes in.
[31:13.020 --> 31:15.680]  So again this is something that not everybody is doing.
[31:15.880 --> 31:17.100]  I think Trevor is in the audience somewhere.
[31:17.280 --> 31:19.640]  This isn't actually a certificate pinning image.
[31:19.760 --> 31:20.900]  I stole it from the TAC website.
[31:21.320 --> 31:32.760]  But the way pinning works is that you rely on the operating system, so the code beneath you, to do all the things that normally happen during SSL handshake.
[31:32.940 --> 31:34.980]  So check out the certificate that you got.
[31:35.160 --> 31:36.600]  Does it match the host name?
[31:36.940 --> 31:37.880]  Has it expired?
[31:38.000 --> 31:38.700]  Has it been revoked?
[31:38.880 --> 31:40.820]  Do all that kind of stuff that you normally do.
[31:41.280 --> 31:49.360]  Then when you're in Appland, check out the certificate that was actually used as part of this SSL handshake.
[31:49.540 --> 31:59.600]  So what you can do if you own the client code, and we do do this in like Twitter for iOS and Twitter for Android, Chrome actually does it for us too, is that we check against the pinned roots.
[31:59.760 --> 32:04.660]  So we say we know that we're only ever going to buy certificates from these three CAs.
[32:05.040 --> 32:11.300]  And if you get a certificate that came from Turk Trust or the Indian government, something's up.
[32:11.500 --> 32:16.660]  Even though this is a certificate you would normally trust, how about you don't trust it in this twitter.com session.
[32:17.320 --> 32:18.900]  And after that, everything else works.
[32:19.060 --> 32:28.140]  It's just kind of like on bootstrapping the SSL connection that you implement the certificate pinning, and nothing else really needs to change in your application.
[32:28.460 --> 32:36.260]  So OWASP, I think, has a great wiki page on implementing cert pinning in your own applications, and so go check that out.
[32:36.380 --> 32:39.420]  It's a little more detailed than I can get to in slides for sure.
[32:41.080 --> 32:43.460]  Of course, security is a cat and mouse game.
[32:44.080 --> 32:49.660]  We rolled out certificate pinning, and then next year, black hat, this talk comes out about getting around certificate pinning.
[32:49.900 --> 32:59.660]  And so the thing I want to stress when it comes to pinning is that the threat we're trying to protect against and the adversary in this situation isn't you yourself with your jailbroken iPhone.
[32:59.880 --> 33:12.680]  So these attacks, while they work, they basically, you know, rooted your device, hooked some DLLs, and then modify the way in which the code was running on the client so they could get around pinning.
[33:12.820 --> 33:20.800]  So this is good if you want to, like, pen test Twitter or you're scared that we're sending some information off your device that you don't want us to send and you want to man-in-the-middle of it.
[33:21.060 --> 33:21.800]  That's cool.
[33:22.040 --> 33:22.840]  Like, do your thing.
[33:24.740 --> 33:25.460]  Intercept the way.
[33:25.740 --> 33:32.760]  But what we don't want to have happen is basically on that first diagram where we have all these hops along the way in between you and us.
[33:32.880 --> 33:35.300]  We don't want anybody there messing with the traffic.
[33:35.480 --> 33:41.560]  So if you want to mess with your own local instance of the application running, go to town, go nuts.
[33:41.720 --> 33:43.820]  That's not the threat we're defending against with pinning.
[33:44.280 --> 33:50.840]  What we want to stop is anybody that's got their hands on a certificate that wasn't us and detecting that.
[33:52.640 --> 34:00.380]  Unfortunately, step one there was rely on the operating system for the first part of the handshake.
[34:00.680 --> 34:04.340]  And this is a slide that I did update this year.
[34:05.560 --> 34:08.340]  This might ring a bell to some folks.
[34:08.580 --> 34:18.520]  But Apple had a kind of nasty bug that came from this, like, second conditional not really being conditional and always jumping down here.
[34:18.980 --> 34:27.120]  Adam Langley, again, being a super legit and super smart guy, has a way better write-up than I could talk through on stage at his blog.
[34:27.620 --> 34:28.280]  Check that out.
[34:28.440 --> 34:39.000]  But basically, if you're relying on the OS for some part of the handshake and you're not doing it yourself, you can be susceptible to the underlying problems in that code.
[34:39.280 --> 34:42.720]  So, Twitter for iOS was affected by go to fail.
[34:43.460 --> 34:47.260]  But we also inherited the patch as soon as iOS patched.
[34:47.420 --> 34:52.140]  So, we could go many steps further than that.
[34:52.280 --> 34:55.800]  But you need to take a reliance on the operating system at some point.
[34:55.940 --> 35:00.680]  Like, we're not going to implement our own full HTTP stack and write all our own TLS code.
[35:00.680 --> 35:02.120]  Like, at some point, you need to draw the line.
[35:03.100 --> 35:05.680]  Pinning is going a lot deeper than most people do.
[35:06.080 --> 35:08.920]  But it still opens up to this kind of problem.
[35:11.380 --> 35:15.080]  So, perfect forward secrecy has been getting a lot of attention lately.
[35:15.560 --> 35:17.240]  And kind of rightfully so.
[35:17.740 --> 35:19.940]  I prefer the term forward secrecy.
[35:20.880 --> 35:24.180]  Perfect is... I've been in security for too long to call anything perfect.
[35:24.580 --> 35:37.280]  But it can be super easy for you to implement if you're just running, like, your own web server and you're not trying to do everything with, like, massive scale and TLS session resumption.
[35:37.360 --> 35:54.380]  Basically, you can go to Mozilla's page for, like, setting up a secure web server, copy and paste the recommended Cypher Suite list that they have for you to implement, and just put that into your HTTP config, and restart your server, and boom!
[35:54.480 --> 36:02.080]  You've got PFS Cypher Suites to the top of the list, and everything that's negotiated from that point forward will use perfect forward secrecy.
[36:02.240 --> 36:05.340]  It will use ECDHE and knock yourself out.
[36:05.460 --> 36:12.580]  If somebody steals your keys, they can't go back and decrypt all the traffic that they may be captured in their mass surveillance efforts.
[36:12.940 --> 36:14.580]  So, forward secrecy is great.
[36:14.700 --> 36:26.220]  We have this blog post because, again, Adam Langley has a blog post about how you can botch forward secrecy if you're using TLS session resumption, which you might want to use for performance reasons.
[36:26.480 --> 36:32.820]  So, we have this post about how you can encrypt your tickets and share them all.
[36:33.060 --> 36:36.660]  Again, check it out because it's a lot more detailed than I can get into here.
[36:36.820 --> 36:40.040]  I think Jacob, who wrote this post and is now at the EFF, is here somewhere.
[36:40.920 --> 36:41.720]  He's a cool dude.
[36:41.900 --> 36:47.900]  I think he might be talking somewhere, but if you see him, say what's up and say thanks for the forward secrecy of Twitter.
[36:51.180 --> 36:52.800]  So, Heartbleed happened as well.
[36:53.040 --> 36:56.640]  And so, sometimes you're good and sometimes you're lucky.
[36:57.420 --> 37:00.740]  And so, Twitter, in this case, got kind of lucky.
[37:00.860 --> 37:07.780]  Our stack is such that there's code that runs before you get to the OpenSSL library.
[37:07.980 --> 37:10.980]  It was essentially too dumb to pass along the Heartbeat request.
[37:11.240 --> 37:18.140]  And so, even if we were running, which we were, vulnerable versions of OpenSSL, the exploit couldn't get to the OpenSSL code.
[37:18.320 --> 37:19.940]  And we were like, oh, my God, we got super lucky.
[37:20.780 --> 37:26.280]  But, all that being said, you know, we rekeyed, we patched the OpenSSL version and all that.
[37:26.420 --> 37:36.960]  And it was the thing that I would like to take away from Heartbleed and pass along to all of you is that it's really good to get good at patching OpenSSL these days from an operational perspective.
[37:37.220 --> 37:37.280]  Right?
[37:38.880 --> 37:42.540]  CCS injection happened, I don't know, like a month and a half later or something.
[37:43.020 --> 37:45.660]  And we learned a whole bunch during Heartbleed.
[37:45.740 --> 37:47.000]  It's like, oh, my God, we have servers over there?
[37:47.100 --> 37:48.340]  They're running what version of SSL?
[37:49.080 --> 37:56.200]  And, you know, over the course of like a few days, we kind of got our act together in, well, I mean, that's harsh.
[37:56.720 --> 37:57.760]  Our entire act together.
[37:57.900 --> 38:00.660]  Stuff was patched super quickly on anything that really, really mattered.
[38:00.920 --> 38:06.820]  But we got to operationalize patching OpenSSL really pretty good by the time the CCS injection came around.
[38:07.100 --> 38:11.580]  And so, it would be silly to assume that, like, that was the last OpenSSL bug.
[38:11.680 --> 38:12.860]  And now we've got it all figured out.
[38:13.140 --> 38:15.440]  So, you should, like, again, practice.
[38:15.680 --> 38:31.500]  Even if there's no CVE that came out against OpenSSL, why don't you go ahead and, like, just pretend there were, restart out your servers, make sure everything is cool and you can still resume your sessions, nothing breaks, and just go ahead and practice as if Heartbleed 2 is right around the corner because it probably is.
[38:33.740 --> 38:35.720]  And with that, I know I've covered a whole bunch.
[38:35.940 --> 38:37.080]  I've got time for questions.
[38:37.600 --> 38:45.160]  And when I give this talk, this is normally the first question, so I wanted to get in front of it and also bring back one of my favorite memes, which is kind of, like, died off on the Internet.
[38:45.700 --> 38:47.560]  But what about performance?
[38:47.760 --> 38:56.780]  And so, what I can say, and this has been documented in other places now, too, is that Twitter is faster now, like, significantly faster, once we move to HTTPS all the time.
[38:56.860 --> 39:08.480]  And there is no way that your code up and down the stack on the client and on the server is so finely tuned that the one thing that you can do to shave off a few milliseconds here and there is use HTTP.
[39:09.440 --> 39:15.980]  So, you kind of just treat it as a non-compromise issue and say, that's it, we're using transport layer security all the time.
[39:17.260 --> 39:27.780]  And find other places for you to speed things up, and you will, and you will get much better gains than you might if you just focus on, even just, like, tweaking your TLS implementation the whole time.
[39:28.700 --> 39:32.720]  You will find more lucrative spots in your code base to fix things up.
[39:33.900 --> 39:35.840]  And with that, I know I ended a little early.
[39:35.960 --> 39:38.040]  I think I've got 10 or 20 minutes left.
[39:39.160 --> 39:41.480]  But this wasn't just the whole pitch to say, oh, come work at Twitter.
[39:41.700 --> 39:41.780]  Awesome.
[39:42.500 --> 39:44.960]  But you can if you want, because we've got a lot of security jobs open.
[39:46.420 --> 39:49.920]  But basically, like I said, this is a whole bunch of stuff we did that happened under the hood.
[39:50.100 --> 39:55.160]  And unless you're super into security and you're sniffing all your traffic all the time, you might not notice that we did it.
[39:55.700 --> 40:01.600]  And that in itself is kind of a good thing, because we changed a lot of what was happening underneath the hood, and people didn't notice.
[40:01.740 --> 40:02.240]  Things didn't break.
[40:02.460 --> 40:11.440]  And so, if we can do it at Twitter, which is humongous, we've got a ton of traffic coming in and out of this place, you can do it on your site, too, and you should do so.
[40:12.240 --> 40:12.860]  And that's it.
[40:13.220 --> 40:13.400]  Thank you.
[40:25.380 --> 40:27.920]  So, you mentioned a couple of things that involve revocation.
[40:28.180 --> 40:28.500]  Yeah.
[40:28.880 --> 40:29.440]  Of certificates.
[40:29.760 --> 40:31.140]  So, like, Heartbleed is a good example.
[40:32.200 --> 40:33.700]  What are you guys doing for revocation?
[40:33.800 --> 40:38.160]  Are you using OCSP stapling, using live OCSP, or are you just relying on CRL sets in Chrome?
[40:38.500 --> 40:38.720]  Yeah.
[40:38.960 --> 40:40.520]  So, revocation, we basically...
[40:40.520 --> 40:42.240]  So, we do nothing fancy for revocation.
[40:42.480 --> 40:45.860]  There was talk about OCSP stapling, and a lot of people push that.
[40:46.420 --> 40:49.660]  We've experimented with it and just haven't implemented it fully.
[40:50.180 --> 40:52.780]  So, basically, same thing for pinning and all that.
[40:52.860 --> 40:57.140]  We just rely on the OS, we rely on the browsers, and kind of have it be up to them.
[40:57.280 --> 40:58.780]  So, if they want to use CRLs, they use CRLs.
[40:58.800 --> 41:02.520]  If they want to use OCSP, use OCSP, but we don't do anything special.
[41:04.960 --> 41:05.780]  Light is killing me.
[41:12.540 --> 41:13.240]  And I'll be around.
[41:13.420 --> 41:14.760]  There are a bunch of other tweets here.
[41:15.140 --> 41:16.480]  Yann up in the front row.
[41:18.120 --> 41:18.520]  Isabella.
[41:18.780 --> 41:20.200]  J.J., you're relatively new, right?
[41:22.000 --> 41:22.740]  Oh, that's right.
[41:22.840 --> 41:24.140]  I've been on Skype with you before.
[41:26.180 --> 41:26.960]  Yeah, what's up?
[41:28.640 --> 41:29.800]  Like I said, I can't see anything.
[41:29.940 --> 41:30.500]  I'm completely blind.
[41:32.400 --> 41:32.400]  Yeah.
[41:32.460 --> 41:33.180]  You said you
[41:41.680 --> 41:42.400]  Economics of what?
[41:44.160 --> 41:46.120]  Oh, yeah, yeah, yeah.
[41:46.280 --> 41:56.800]  So, the question, yeah, the question was that, I mentioned that we pin to basically, like, three different root certificates, and what are the economic implications of doing so?
[41:56.920 --> 41:59.560]  So, you're right that there can be.
[42:00.000 --> 42:05.780]  There's a little bit of a good faith in there that says, like, okay, now that we've pinned to VeriSign, VeriSign is going to say, oh, it's time to renew the certs, huh?
[42:06.080 --> 42:10.100]  $400 million, because we're pinned in that way.
[42:11.600 --> 42:12.560]  Dr. Evil and what not.
[42:13.840 --> 42:26.920]  But so, the reason we have multiple is basically in the case where somebody does stick it to us, potentially, we can have a little bit of agility, and we can also upgrade the clients over time to change out any roots if we need to do that.
[42:28.120 --> 42:31.520]  And then, the multiple certs are there as well as kind of like defense in depth.
[42:31.680 --> 42:37.380]  So, say that we rely on VeriSign, VeriSign, VeriSign issues of bad certificate, and we need to revoke that.
[42:37.560 --> 42:39.200]  We can fall back to one of the other two.
[42:40.320 --> 42:46.920]  One thing to note about pinning, and we've seen bad pinning implementations happen for sure, where people pin against the leaf certificate.
[42:47.200 --> 42:49.720]  So, this is a certificate that expires every year.
[42:50.200 --> 43:02.980]  And, you know, when April rolls around, all of a sudden, all of these clients, I won't name any names, stop working because they pinned to basically a SHA hash of the leaf cert when you should pin to the root.
[43:03.240 --> 43:06.840]  So, it's a little gotcha for anybody implementing their own pinning.
[43:11.710 --> 43:13.610]  Hey, follow up to that one.
[43:14.050 --> 43:19.190]  Why do you pin to roots rather than using an intermediate offline signing key for yourself?
[43:19.190 --> 43:19.910]  Yeah.
[43:20.270 --> 43:24.470]  So, it's kind of just the ease of implementation, I think.
[43:25.530 --> 43:31.150]  You're right that we could just roll out our own entire, basically, API endpoint that's signed by ourselves.
[43:31.290 --> 43:32.130]  We could run our own CA.
[43:32.910 --> 43:36.370]  This is sort of easy because you get a bit of the best of both worlds.
[43:36.370 --> 43:47.630]  You can serve up normal traffic to normal people with their own root store, and they can trust us because they trust the CAs versus, like doing an intermediate cert on the side.
[43:47.790 --> 43:51.870]  So, there's a TAC proposal which is out there which allows for this kind of thing.
[43:52.570 --> 43:57.330]  It's been a while since I've looked at the proposal, but it was something that Moxie and Trevor worked on.
[43:58.730 --> 44:03.090]  For it, we just need more like browsers to pick it up too, right?
[44:03.150 --> 44:06.210]  There's kind of takes two to tango kind of thing, right?
[44:06.290 --> 44:07.990]  Like you need the client and the server to agree on it.
[44:08.130 --> 44:09.450]  And so, we just need some more adoption.
[44:09.750 --> 44:15.490]  And we look at it, but without something like that, we just kind of like use something that we can use most widely.
[44:17.330 --> 44:17.730]  Thanks.
[44:20.990 --> 44:22.630]  I can't tell... Yeah, what's that?
[44:30.460 --> 44:32.580]  Clients to use SSL links?
[44:33.960 --> 44:35.340]  Not SSL links, right?
[44:35.420 --> 44:42.920]  Because we don't want every person that wants to share a link on Twitter to have to like register SSL for their domain.
[44:43.220 --> 44:43.940]  So, what we do...
[44:48.020 --> 44:48.920]  Exactly, exactly.
[44:49.160 --> 44:50.380]  So, there was some talk early on.
[44:50.380 --> 44:53.080]  So, we run TCO, which is our own link shortening service.
[44:53.720 --> 45:04.860]  And what we do for kind of our TCO policy is if you tweet an HTTP link, so HTTP hope.net, we'll give you an HTTP t.co shortening version of the URL.
[45:05.180 --> 45:10.040]  And so, there was some talk about like, oh, we should just use SSL all the time on TCO.
[45:10.380 --> 45:15.240]  And in that case, you would lose some of the refer information and people would get kind of bummed out about that.
[45:15.360 --> 45:21.360]  And it's just unnecessary overhead for us in a while in a ways because you end up on HTTP URL anyway.
[45:21.580 --> 45:26.440]  So, you tweet an HTTP link and then we move it to SSL, but then we redirect you back to HTTP.
[45:26.680 --> 45:28.100]  We don't really gain too much about that.
[45:28.220 --> 45:34.540]  And sorry, I didn't repeat the question there, but it was like, do you force advertisers to use HTTPS links on Twitter?
[45:35.700 --> 45:37.020]  No, because of that.
[45:37.260 --> 45:41.720]  If you're like a first party ad server, we're moving more and more.
[45:41.840 --> 45:48.280]  So, I mean, we run our own ad network too, but if you want to embed some ad content, we're putting...
[45:53.750 --> 45:58.250]  and we are actively moving our partners over to SSL all the time.
[45:58.950 --> 45:59.370]  Yep.
[45:59.530 --> 45:59.670]  What's up?
[46:00.110 --> 46:02.510]  You mentioned the three CAs that you trust.
[46:02.790 --> 46:04.150]  Which three do you trust?
[46:05.290 --> 46:07.390]  I think it's DigiCert...
[46:13.440 --> 46:15.140]  Everybody kind of becomes a rare sign.
[46:15.920 --> 46:17.760]  And that can be a promise.
[46:17.960 --> 46:18.440]  Not everybody.
[46:18.880 --> 46:19.200]  Yeah, sure.
[46:19.300 --> 46:19.600]  Sure, sure.
[46:19.740 --> 46:19.840]  Yeah.
[46:20.460 --> 46:21.260]  Keep fighting the good fight.