[00:09.640 --> 00:11.420]  And subsets of videos.
[00:14.000 --> 00:16.180]  Make sure they're all cued to where I want them.
[00:31.170 --> 00:31.770]  Yep.
[00:35.090 --> 00:35.690]  Yep.
[00:46.490 --> 00:48.610]  Alright, I think I'm completely cued.
[00:49.070 --> 00:49.210]  Alright.
[00:52.410 --> 00:55.090]  So, will you flash me a five minutes?
[00:55.310 --> 00:55.310]  Five minutes.
[00:55.450 --> 00:55.510]  Yep.
[00:56.390 --> 00:56.750]  So...
[00:56.750 --> 00:58.490]  We don't have signs in this room.
[00:58.590 --> 00:59.710]  Someone stole the signs.
[00:59.990 --> 01:01.690]  So I'm going to have to use my hand.
[01:03.230 --> 01:04.230]  And where are you sitting?
[01:04.530 --> 01:05.290]  I'll be right there.
[01:05.550 --> 01:05.930]  Oh, okay.
[01:06.110 --> 01:06.270]  Good.
[01:06.470 --> 01:06.810]  That's easy.
[01:07.350 --> 01:08.310]  Make sure you sit here.
[01:08.790 --> 01:08.990]  Okay.
[01:10.590 --> 01:12.270]  As usual, I'm trying to cram too much in.
[01:12.390 --> 01:13.230]  Let's see what we can do.
[01:13.390 --> 01:13.490]  Yep.
[01:16.340 --> 01:18.620]  Okay, it's now the crack of noon.
[01:18.620 --> 01:20.780]  So we can get started.
[01:21.380 --> 01:23.660]  A couple of very quick announcements.
[01:24.540 --> 01:30.900]  In case you haven't heard, turn your damn cell phone to stun, please.
[01:30.900 --> 01:32.000]  Thank you.
[01:32.960 --> 01:36.780]  There's still time to sign up for a couple of lightning talk slots in the U room.
[01:36.960 --> 01:39.840]  If you have not already done so, you want to do that.
[01:40.460 --> 01:44.840]  Also, this session will be followed by one of our keynotes, Daniel Ellsberg.
[01:44.840 --> 01:49.180]  I would suggest if you want to see it, it will be simulcast in all the rooms.
[01:49.440 --> 01:54.720]  You are best off keeping yourself here rather than trying to move after this talk.
[01:55.580 --> 01:58.820]  We will have a hard stop as a result of that at...
[01:58.820 --> 02:00.800]  Don't adjust it for me because I'm running right off.
[02:01.540 --> 02:01.620]  Yeah.
[02:02.280 --> 02:06.700]  We're going to have a hard stop at 55 minutes because of that.
[02:06.700 --> 02:12.140]  So we're going to be extra strict with ending the session.
[02:12.140 --> 02:29.020]  That said, it is my great, great pleasure to introduce a terrific speaker expert in visualization who I have known for many years and was just unbelievably happy to get to come to HOPE.
[02:29.200 --> 02:33.200]  I had to promise to buy her lots of good sushi and it will be well worth it.
[02:33.660 --> 02:43.940]  Tamara Munzner is a professor at the University of British Columbia and an expert in visualization and she has an amazing talk and you're lucky to see it.
[02:44.160 --> 02:45.220]  So let me turn it over to Tamara.
[02:45.580 --> 02:45.940]  Tamara Munzner.
[02:50.660 --> 02:51.240]  All right.
[02:51.460 --> 02:53.100]  Thanks for that incredibly kind introduction.
[02:53.360 --> 02:55.980]  The part that is true is I've been doing this for over 20 years.
[02:56.180 --> 02:57.740]  So I've learned a few things along the way.
[02:58.320 --> 03:03.000]  And so today I'm going to talk about visualization for hackers in particular, why it's tricky and where to start.
[03:03.380 --> 03:05.960]  And so an intro, what is viz?
[03:06.060 --> 03:07.420]  This word I'm tossing around.
[03:07.740 --> 03:10.140]  I'm going to give two examples that we're going to walk through.
[03:10.140 --> 03:13.820]  One is a system for looking at server logs for managed web hosting.
[03:13.980 --> 03:15.800]  And this was done in collaboration with folks at AT&T.
[03:16.120 --> 03:17.940]  And the second project is called Overview.
[03:18.280 --> 03:22.700]  And this is for semi-structured text, which is visual document mining for journalists.
[03:23.180 --> 03:25.960]  And that was done in collaboration with folks from the Associated Press.
[03:26.160 --> 03:32.840]  And then I'll try to take what I've talked about and fit it into a bigger picture and wrap things up all in the nick of time by 55 minutes too.
[03:33.960 --> 03:36.280]  So let's start out with defining visualization.
[03:36.280 --> 03:39.600]  I'm going to say viz to save many, many syllables over the course of this next hour.
[03:40.340 --> 03:47.300]  So computer-based visualization systems provide visual representations of data sets designed to help people carry out tasks more effectively.
[03:47.320 --> 03:49.460]  That's my personal favorite definition these days.
[03:49.740 --> 03:51.460]  So, so what?
[03:51.580 --> 03:52.240]  Why do we care?
[03:52.320 --> 03:54.140]  What are the implications of this definition?
[03:54.960 --> 03:56.780]  Also, I'm going to point this down a bit.
[03:57.020 --> 03:57.340]  There.
[03:58.480 --> 04:00.800]  So one crucial thing is there's a human in the loop.
[04:00.880 --> 04:02.020]  There's data and there's people.
[04:02.180 --> 04:03.400]  And when do we need this?
[04:03.400 --> 04:12.020]  Well, in particular, we need viz when there is some requirement that we want to augment the capabilities of a human instead of just replace them with a computer program.
[04:12.180 --> 04:14.120]  But of course, we're all a bunch of computer people here.
[04:14.140 --> 04:15.440]  So when would we do that?
[04:15.600 --> 04:15.840]  Why?
[04:16.660 --> 04:20.760]  The problem is that if you know exactly what you're writing a program for, you can do it.
[04:20.860 --> 04:27.140]  But what if you have one of hundreds or tens of thousands of questions you might ask and you don't know in advance exactly what you want?
[04:27.140 --> 04:30.100]  That's typically a situation where visualization is useful.
[04:30.340 --> 04:39.060]  If you've got something that's fully automatic, that has been written, and that you actually trust, those are two necessary and sufficient conditions, then you don't need viz.
[04:39.280 --> 04:40.980]  But there's a lot of situations where you do.
[04:41.260 --> 04:42.320]  So I'm curious.
[04:42.320 --> 04:46.600]  How many people have actually seen Anscombe's Quartet before in the audience?
[04:48.220 --> 04:48.820]  One.
[04:48.820 --> 04:49.680]  All right.
[04:49.900 --> 04:55.120]  So this is a great set of four data sets designed in 1973 by Anscombe, a statistician.
[04:55.260 --> 05:00.800]  And these four data sets have identical mean and variance in X and in Y and regression lines.
[05:01.000 --> 05:04.180]  So according to this nice statistical summary, we can't tell them apart.
[05:04.780 --> 05:08.740]  The instant I show you these pictures, you can tell them apart, right?
[05:09.000 --> 05:12.100]  One's got what you might think of as the right behavior for these stats.
[05:12.240 --> 05:13.500]  One's obviously nonlinear.
[05:14.060 --> 05:16.600]  One, the slope line's a bit off because of an outlier.
[05:16.600 --> 05:19.960]  We've got the terrible, pessimal case where the slope line is deeply misleading.
[05:20.360 --> 05:22.240]  And the point is that's 20 dots.
[05:22.760 --> 05:25.420]  What about when you've got hundreds of thousands of data points?
[05:25.420 --> 05:27.240]  It's dynamic data changing over time.
[05:27.440 --> 05:31.180]  There's a lot of situations where human pattern recognition can actually be very useful.
[05:31.460 --> 05:36.460]  And you want to employ that as part of your analysis process.
[05:38.520 --> 05:45.000]  So the idea of having these visual representations is that what we're trying to do is get stuff out of your head and outside.
[05:45.000 --> 05:49.660]  So more generically, I'm going to call that an external representation, to use a word out of cognitive psychology.
[05:50.060 --> 05:55.400]  And what we're trying to do in a bunch of situations is replace something that's cognitive with something that's perceptual.
[05:55.700 --> 05:57.040]  So let me give you an example.
[05:57.220 --> 05:58.660]  You've got a bunch of time series data.
[05:58.660 --> 06:00.340]  We're going to be looking at this later in the talk.
[06:00.500 --> 06:01.760]  So you've got time points.
[06:01.980 --> 06:02.860]  You've got numbers.
[06:02.860 --> 06:05.760]  Now, if I ask you things like, what's the trend here?
[06:06.260 --> 06:08.300]  You can figure that out given enough patience.
[06:08.500 --> 06:10.060]  But what you're doing is you're reading a number.
[06:10.260 --> 06:12.940]  You're storing it in your head, an internal mental buffer.
[06:13.120 --> 06:14.380]  You're reading the next number.
[06:14.480 --> 06:16.020]  You're doing a mental comparison.
[06:16.300 --> 06:18.700]  And all of these are possible, but they take load.
[06:18.920 --> 06:24.400]  And if you have enough cognitive load, you can't do other things, like actually solve the higher level problem that you're looking at.
[06:24.400 --> 06:38.440]  So if, in contrast, we do something straightforward, like let us take these numbers and encode them as a picture, and we'll be talking about visual encoding as we go, then we've replaced something cognitive that demands memory and attention with something that we can do directly with our perceptual systems.
[06:38.600 --> 06:42.600]  And we've got a bunch of hardwired background processing in our brains for doing that.
[06:43.020 --> 06:46.200]  If this were another talk, I'd go much more into detail about the mechanisms for that.
[06:46.380 --> 06:49.300]  In this one, I'll talk more about the implications of those mechanisms.
[06:50.840 --> 06:54.960]  So, what I'm going to present today is an analysis framework for thinking about this systematically.
[06:55.240 --> 07:06.560]  For those of you who were just in the room for Ella's talk on threat modeling, there's been a bunch of work in visualization where we can do something similar, where we can actually have a model of how to think about visualization in a structured way that helps you get things done,
[07:06.660 --> 07:08.920]  rather than random walks through the parameter space.
[07:09.740 --> 07:12.700]  So, I'm going to focus on four levels that you can think about.
[07:12.820 --> 07:15.840]  What I'm going to call the domain, in this case, you guys are domain experts.
[07:15.840 --> 07:19.980]  The domain is some particular either domain or subdomain.
[07:20.200 --> 07:23.640]  You've got someone who is an actual target user with some problem.
[07:24.320 --> 07:26.300]  So, typically, they've got data, they've got problems.
[07:26.680 --> 07:30.380]  Now, what I'm going to call the abstraction level is an interesting one.
[07:30.540 --> 07:39.140]  In particular, what I want to do is translate from some domain-specific vocabulary to something that we can actually design visualizations to help with.
[07:39.280 --> 07:46.700]  And that's how we can actually learn things across from hacking to biology to financial modeling to looking at your music listening history.
[07:47.640 --> 07:51.700]  I'm going to focus on two different questions and expand out this idea of the abstraction level.
[07:51.920 --> 07:53.640]  First of all, what are we showing?
[07:53.800 --> 07:55.280]  That's the data abstraction.
[07:55.860 --> 07:58.120]  And why is the user looking at it?
[07:58.240 --> 07:59.500]  That's the task abstraction.
[07:59.720 --> 08:03.740]  So, these words, data and task, I will be hammering on repeatedly over this next hour.
[08:04.880 --> 08:10.640]  Now, I'm going to call the idiom level is basically, you might hear, called a technique or a system or a method.
[08:11.140 --> 08:15.080]  And it's the question of how do we show this on the screen.
[08:15.520 --> 08:19.720]  And once again, I'm actually going to subdivide and I'm going to talk about visual encoding.
[08:19.940 --> 08:21.680]  And that is specifically what to draw.
[08:21.940 --> 08:24.420]  This is something you could even do with something completely static.
[08:24.800 --> 08:28.600]  And also the interaction idiom, which is how to manipulate that picture.
[08:28.820 --> 08:35.640]  So, much of the interesting parts of visualization are the interactivity, which is what we've got now that we've got computer-based visualization.
[08:36.460 --> 08:46.820]  And in particular, I'm separating these design concerns of the idiom, how to draw and how to manipulate from the algorithm level of how do you compute that efficiently.
[08:47.080 --> 08:54.260]  Because a lot of people have a tendency to dive into the algorithm level instantly without understanding what exactly is the right thing to be building.
[08:54.560 --> 08:57.140]  We all know that premature optimization is the root of all evil.
[08:57.740 --> 09:02.820]  Algorithmic work before you understand the right idiom and abstractions and viz is exactly that evil.
[09:04.780 --> 09:07.280]  So, why do this kind of analysis?
[09:07.820 --> 09:14.760]  Well, the issue is there's actually an enormous design space for how to design these kinds of interactive visualizations.
[09:15.020 --> 09:27.960]  Just the visual encoding alone, this question of how do you have marks on a display that actually communicate information using things like size and shape and color and position, that's a very big combinatorial explosion.
[09:27.960 --> 09:31.360]  And then when you add interactivity, the space is bigger.
[09:31.640 --> 09:38.360]  And then, here's an interesting part, with the data abstraction, you don't always just draw the data you're given.
[09:38.660 --> 09:44.660]  You often transform it to a form such that then when you encode that, you've got something useful.
[09:44.860 --> 09:48.800]  So there's a transformation possibility that makes that design space even bigger.
[09:50.500 --> 09:55.720]  So the other crucial but tragic fact is that most of the possibilities don't work.
[09:55.900 --> 10:07.500]  In particular, if you've got a combination of task and data, there's going to be a lot of choices that are much less useful than others, in part because of the way our brains are structured in terms of how we interpret visual information.
[10:08.200 --> 10:12.220]  So, what that means is random walks are a sad thing we want to avoid them.
[10:12.580 --> 10:16.200]  Some kind of guidance of principles can actually help you be a lot more effective.
[10:16.920 --> 10:23.760]  And so, this analysis framework I'm going to talk about is a scaffold to let you actually think systematically about this design space.
[10:25.400 --> 10:40.480]  What I want to get people to do is to actually think broadly enough about the possibilities that they treat this in the right way for design problems, where what you want is to consider a very large set because the thing is, as in many design problems,
[10:40.500 --> 10:43.680]  we're not just optimizing for a single perfect best solution.
[10:43.680 --> 10:47.300]  There is a fairly large set of good answers.
[10:47.700 --> 10:52.160]  We want to get one of those, as opposed to the truly enormous set of bad answers, which we're hoping to avoid.
[10:52.420 --> 11:00.620]  And by understanding what this set of possibilities are in the Viz design space, we're going to maximize the chances that you'll actually get one of these reasonable ones.
[11:01.740 --> 11:07.300]  Today, I'm going to focus more on this abstraction and idiom level as things that are maybe less familiar to you folks.
[11:07.740 --> 11:15.560]  I'll dive into the algorithm level occasionally because I just can't help myself because they're fun, but mostly we're going to try to stay up at abstractions and idioms.
[11:17.620 --> 11:27.540]  Okay, so with that intro, I'm now going to blast onward and actually, instead of presenting the whole framework all at once, I'm going to talk you through systems and discuss some of these design choices.
[11:27.540 --> 11:29.720]  So I'm going to start with this LiveRack system.
[11:30.020 --> 11:41.180]  This is done jointly with Peter McLaughlin, Lefty Kudsofios, and Steven North, the latter two at AT&T, who have big collections of time series data from systems management.
[11:42.180 --> 11:46.940]  I'm going to start out with a video to show you the look and feel, not only of the visual encoding, but also the interaction idiom.
[11:47.320 --> 11:49.180]  I'm just going to show a snippet of this video.
[11:49.480 --> 11:51.600]  The whole one, the link is available there.
[11:52.220 --> 11:53.860]  So, let's run...
[11:53.860 --> 11:58.080]  LiveRack, interactive visual exploration of system management time series data.
[11:59.800 --> 12:06.080]  We show LiveRack, a visualization system for viewing large quantities of time series data in the domain of system management.
[12:06.700 --> 12:08.900]  Monitor devices are represented by rows.
[12:09.200 --> 12:13.600]  This label shows aggregate information about a group, NFS, with four devices.
[12:14.440 --> 12:17.480]  Columns show one or many parameters collected from the devices.
[12:17.920 --> 12:19.640]  This column shows CPU usage.
[12:20.000 --> 12:23.100]  In the current overview, each cell is a small colored block.
[12:23.100 --> 12:27.940]  In these four cells, green means CPU utilization is above 35%.
[12:28.560 --> 12:32.720]  As I make the cells larger, labels for the individual devices become visible.
[12:33.160 --> 12:40.260]  Also, when the cells get big enough, the server is queried for more data, and then the representation changes, from blocks to sparklines.
[12:40.940 --> 12:45.440]  Making a cell even bigger results in semantic zooming, to show a more detailed line graph.
[12:45.440 --> 12:52.780]  We can see the current time range by looking at the time boxes, to the bottom left, and bottom right of the data area.
[12:53.360 --> 12:55.620]  The blue lines show progressive search results.
[12:55.980 --> 13:00.040]  I'm searching on NFS, to flag the devices I've already stretched out.
[13:03.980 --> 13:05.680]  Interpreting network environment state.
[13:08.080 --> 13:13.580]  I'm acting in the role of a network administrator, interested in looking at CPU utilization of the top few offenders.
[13:14.020 --> 13:20.660]  I begin by sorting the CPU column, and then performing a stretch operation on the top few rows, to obtain more detailed trend information.
[13:21.360 --> 13:23.520]  The sparklines show trend data for one week.
[13:23.800 --> 13:26.420]  The red squares on each trend line indicate a high water mark.
[13:26.860 --> 13:31.080]  I can see a number of the web devices share the same high water mark in the middle of the week.
[13:31.780 --> 13:34.020]  Load, a measure that accounts for I.O.
[13:34.020 --> 13:38.440]  in addition to CPU usage, was also at a peak for this device during this time.
[13:38.800 --> 13:44.020]  But I noticed this was not the case for Scoop, where load peaked early in the week, and CPU later on.
[13:45.180 --> 13:47.620]  I/O load was also not correlated to CPU usage on tweak.
[13:48.120 --> 13:52.840]  We can see a steep drop in the number of processes running on tweak at the exact same time as manipulator.
[13:53.500 --> 13:57.780]  This drop correlates with a drop in memory usage, which freed up some swap memory as well.
[13:58.540 --> 14:08.880]  When I grow the memory in swap cells for tweak, I can see that a small, slow increase in memory usage was enough to cause the swap in the first place, and that it seems to run quite near the threshold where swapping begins.
[14:09.560 --> 14:11.780]  This device can definitely benefit from more memory.
[14:12.660 --> 14:14.300]  Manipulator does not share these characteristics.
[14:14.300 --> 14:24.600]  We can see that memory usage built through the week for manipulator, correlating with the period of heavy CPU utilization and load, but that the memory was properly released when load dropped and no swap was used.
[14:26.180 --> 14:31.420]  I expand my time window to six months to see whether some of these trends extend over a longer period of time.
[14:32.300 --> 14:42.540]  Six months is a large query, so it takes a while to get the new data from the server located on the other side of the country, which stores data for dozens of parameters on thousands of devices across several years.
[14:43.060 --> 14:49.340]  The yellow dots show that the new data hasn't arrived yet, but I can still interactively explore on the visualization client in the meantime.
[14:50.460 --> 14:51.900]  Here is an interesting trend.
[14:52.180 --> 14:54.480]  The utilization on Scoop is trending downwards.
[14:55.040 --> 14:59.360]  We can also see that there was a steep drop in memory usage on several devices in August.
[15:00.180 --> 15:03.280]  Network inbound and outbound traffic also seems to be trending downwards.
[15:04.040 --> 15:07.900]  I might want to consider consolidating some of these devices if these trends continue.
[15:09.260 --> 15:10.700]  Okay, I'm going to stop that there.
[15:11.000 --> 15:13.680]  That's enough to give you a flavor of the system in action.
[15:14.200 --> 15:17.760]  And now what I want to do is talk about it in terms of this what, why, and how.
[15:18.180 --> 15:19.280]  So first of all, what?
[15:19.400 --> 15:21.060]  What is the data abstraction here?
[15:21.760 --> 15:23.700]  And so we've got a multidimensional table.
[15:23.860 --> 15:25.280]  This is one of the most common cases.
[15:25.440 --> 15:27.940]  You've got a giant table of numbers and stuff.
[15:28.160 --> 15:30.620]  In particular, now our numbers are time series data.
[15:31.160 --> 15:36.400]  So we're going to think of this as, when I talk about tables, I'm going to talk about the columns of the tables as attributes.
[15:36.560 --> 15:38.900]  Sometimes these are called dimensions or variables or columns.
[15:39.060 --> 15:39.960]  I'll call them attributes.
[15:40.140 --> 15:44.380]  And then the rows in the table, conceptually, if it's a simple flat table, are items.
[15:44.560 --> 15:49.060]  But of course, in a lot of interesting tables like multidimensional arrays, you've got multiple keys to look things up.
[15:49.060 --> 15:50.780]  So in this case, what have I got?
[15:51.140 --> 15:54.840]  Well, the keys of this table, as we saw, it's time series data, so there's time.
[15:55.140 --> 16:01.000]  We wanted to build something that's scaled to about gathering data at five-minute intervals for around six months.
[16:01.200 --> 16:03.640]  So design target of about 50,000 data points there.
[16:04.280 --> 16:07.180]  Now what's interesting about time is there's multiple levels of interest.
[16:07.400 --> 16:09.820]  So there's a hierarchical structure in time.
[16:09.960 --> 16:14.080]  They might care about things at the level of, say, minutes versus hours, days, weeks, months.
[16:14.240 --> 16:17.440]  So we wanted to support that kind of analysis at these multiple levels.
[16:18.280 --> 16:19.800]  We talked about devices, right?
[16:19.940 --> 16:23.680]  See, these are, in this case, it's a bunch of machines in a server room somewhere.
[16:24.100 --> 16:27.200]  And so we were aiming for thousands as a design target.
[16:27.840 --> 16:32.380]  And then parameters, we saw these were things like we wanted to handle a dozen or two.
[16:32.700 --> 16:36.300]  So CPU usage, memory load, network traffic, alarms.
[16:37.080 --> 16:41.360]  So that was the design target in terms of how to look things up in the table.
[16:41.660 --> 16:42.560]  And then what's in it?
[16:43.140 --> 16:48.520]  Well, there's a lot of numbers, which are just the parameter value for the device at the time point.
[16:49.220 --> 16:56.140]  These are quantitative attributes as opposed to device groups, which are more of a categorical attribute.
[16:56.300 --> 16:57.680]  These devices are together in a group.
[16:57.680 --> 17:01.080]  So these buzzwords here, quantitative and categorical, what are they?
[17:01.440 --> 17:03.680]  Well, there's different kinds of attribute types.
[17:03.840 --> 17:11.080]  And as we'll see coming up, there's a critical distinction about whether what you've got is simply saying it's in this category or it isn't.
[17:11.240 --> 17:12.480]  Sometimes these are called nominal.
[17:13.400 --> 17:17.420]  As opposed to ordered data, where you actually are saying, there's an ordering.
[17:17.600 --> 17:18.820]  I can do arithmetic on them.
[17:19.160 --> 17:23.300]  And so things like, right, you know, a load is 0.8 from a scale of 0 to 1.
[17:23.460 --> 17:26.000]  Then that's something you can actually do measurements on.
[17:26.420 --> 17:33.060]  So just like, again, those of who are in the room for Ella's talk, she talked about a structured vocabulary.
[17:33.360 --> 17:39.320]  Part of what I'm trying to introduce are not only ideas, but a vocabulary for talking about them in a more systematic way.
[17:39.840 --> 17:41.360]  So this is our what.
[17:41.640 --> 17:43.620]  It's actually relatively straightforward as Viz go.
[17:44.120 --> 17:45.200]  How about our why?
[17:45.700 --> 17:49.960]  Now, I mentioned there's this idea of going from domain language to something more generic.
[17:50.200 --> 17:53.100]  So in domain language, what were the people worrying about?
[17:53.340 --> 17:57.660]  This is actually a mix of frontline operations and more forensic analysis.
[17:57.980 --> 18:15.540]  So it was things like interpret the status of the network environment, generate reports, do capacity planning for when to add or get rid of machines, and investigation of events in a forensic way of why did things just go wrong and everything melted on the day that ESPN was running this basketball game.
[18:15.700 --> 18:18.420]  Because AT&T is actually doing managed hosting for some very big firms.
[18:18.820 --> 18:22.020]  And then a lot of coordination between customers and engineering and ops.
[18:22.220 --> 18:24.500]  So this is not a generic statement.
[18:24.640 --> 18:29.420]  This is a domain specific statement that you have to have some kind of domain expertise to figure out where to go from here.
[18:31.440 --> 18:48.180]  So the part I'm leaving out now is many, many, many rounds of going off, talking to them, understanding, building prototype systems, refining that, finally actually getting access to the operations people after convincing their bosses to let us steal some of their time to actually talk about them for what they need.
[18:48.280 --> 18:55.780]  But in the end, what we ended up characterizing this as is that we want to be able to browse and correlate across different combinations of these parameters and devices and times.
[18:55.780 --> 19:11.060]  And so do things like correlate the alarm attribute with these other parameters or find trends across an entire group of devices, summarize what happened over different time intervals, identify devices either at a threshold or beyond it, figure out what those thresholds are,
[19:11.460 --> 19:14.540]  try to compare behavior at specific times across devices.
[19:15.120 --> 19:17.820]  So there's some words that are actually starting to recur.
[19:18.100 --> 19:23.340]  So again, for this idea of a structured vocabulary, well, one thing we're doing is we're trying to support discovery.
[19:23.340 --> 19:26.400]  So this is not just tell people something you know already.
[19:26.580 --> 19:28.280]  This is figure out what's in the data.
[19:28.420 --> 19:30.400]  So it's a discovery task at the high level.
[19:31.620 --> 19:32.760]  Different kinds of actions.
[19:33.620 --> 19:42.540]  In this structured way of thinking about it, I often try to distinguish between identifying one thing, comparing a small set of things, and then summarizing the whole set.
[19:42.740 --> 19:47.340]  So this, do you care about one or some or all, is a crucial question to ask about what you're doing.
[19:47.520 --> 19:50.540]  So if those are actions, well, then what's the target of the action?
[19:50.540 --> 19:55.180]  Some targets have to do with any kind of data set whatsoever, like find trends, find outliers.
[19:55.360 --> 19:56.340]  Those are the classic ones.
[19:56.980 --> 20:03.300]  Some kind of domain-specific thing I'm going to call a feature that maybe you've got some domain-specific definition of.
[20:03.780 --> 20:07.900]  And then there's some targets that have to do with the characteristics of the actual data.
[20:08.100 --> 20:14.520]  So in this case, with my table where I've got these attribute columns, sometimes I want to look at the distribution within an attribute.
[20:14.520 --> 20:19.960]  And sometimes I want to do things that have to do with, say, correlating multiple attributes with each other.
[20:20.880 --> 20:25.820]  So this starts to give us a vocabulary for talking about the actions and targets of the user.
[20:27.360 --> 20:29.420]  So now let's start talking about this question of how.
[20:29.540 --> 20:30.860]  How did we actually design the idiom?
[20:31.040 --> 20:34.140]  Lots of stuff was going on in various flashy and blinky ways in the video.
[20:34.300 --> 20:35.620]  So let's decompose that.
[20:35.620 --> 20:39.040]  So one crucial choice was there were multiple views.
[20:39.220 --> 20:42.080]  There was some way that I had taken the data and I'd faceted it.
[20:42.180 --> 20:46.080]  That is, I'd chopped it up, partitioned it across multiple different views.
[20:46.220 --> 20:49.980]  And in particular, I was juxtaposing these views side by side.
[20:50.380 --> 20:57.080]  So here's one of these small examples where we're seeing a bunch of these sparklines showing CPU across different devices.
[20:58.180 --> 21:02.820]  So what's interesting and important about that, and this gets back to the idea of perception versus cognition.
[21:02.820 --> 21:10.460]  When you look at two things side by side, your eyes can move between them, and you have a very low cognitive load way of making comparisons.
[21:10.780 --> 21:17.100]  If you have one view that changes over time, then you're comparing the memory of what you saw before to the thing that's in front of your eyes now.
[21:17.300 --> 21:18.640]  That is higher cognitive load.
[21:18.840 --> 21:25.720]  So you can start to think of this as a tradeoff space between the amount of room you need on the screen for different windows, and then the amount of user's memory that you're taking.
[21:26.040 --> 21:28.720]  So now we start to think about our set of choices.
[21:29.640 --> 21:33.440]  So in particular, there's different flavors of these multiple view systems.
[21:34.080 --> 21:38.040]  One question is, do you have the same way of drawing in all of the views or not?
[21:38.160 --> 21:39.840]  And then one is, how much of the data?
[21:40.040 --> 21:45.120]  Is it all the data in all the views, or just a subset, or actually completely disjoint partitions?
[21:45.320 --> 21:53.060]  So this case where you've got different data in each view, and you're drawing it the same way, the buzzword envis is often small multiples for this case.
[21:54.760 --> 21:56.100]  So, what else do we do?
[21:56.180 --> 21:59.560]  In order to make multiple views useful, you somehow link them.
[21:59.920 --> 22:03.440]  One way you can link them has to do with whether or not you're actually using the same encoding.
[22:03.760 --> 22:09.360]  But a really classic case is that interactively, when the user does something in one view, you want to see the results in the others.
[22:09.540 --> 22:11.760]  That's giving you the power of these multiple views.
[22:11.880 --> 22:18.280]  And so in this system in particular, whenever you're moving the mouse in one of these little cells, you see a marker line tracking your exact location.
[22:18.280 --> 22:21.040]  So you can do high precision comparisons in the other views.
[22:21.040 --> 22:24.600]  And there's a lot of different ways of linked highlighting, some of which I'll talk about a bit later.
[22:26.680 --> 22:31.440]  Now, there was this interesting and weird interaction metaphor that I called semantic zooming.
[22:31.660 --> 22:46.340]  It's a general class of things, where as you think about navigation, instead of just doing something geometric, where what you see gets bigger or smaller, just like a piece of paper would if you brought it closer and further to your eyes, there's some interesting ways of representing where you say,
[22:46.380 --> 22:50.620]  I'm going to dramatically change what I draw based on the number of pixels available to me.
[22:50.620 --> 22:55.920]  So let me do something smart, not just draw something with a bunch of things smoshed together when the area gets too small.
[22:56.260 --> 22:59.440]  Let's go from the complex case, if we've got enough pixels, great.
[22:59.640 --> 23:04.200]  We'll have superimposed line charts, and they'll have labels, and they'll have axes, and you can do high precision things.
[23:04.360 --> 23:05.840]  And then at some point, we give up on that.
[23:06.020 --> 23:13.560]  If we don't have enough pixels, we draw these very iconic representations with just a single line, and then a few dots to show things like high and low marks.
[23:13.800 --> 23:19.240]  And if we really don't have very many pixels, if I just give you a few pixels, don't even try to draw that line.
[23:19.420 --> 23:21.100]  Just draw a color-coded box.
[23:21.380 --> 23:23.580]  That becomes something like a heat map view.
[23:24.620 --> 23:26.420]  So we see examples of all of these here.
[23:27.280 --> 23:29.280]  So again, let's think about this structurally.
[23:29.580 --> 23:35.620]  Well, one thing we're doing with semantic zooming is this is an example of something that's changing dynamically over time.
[23:35.620 --> 23:41.900]  So we've got... and this is the most common form of interactivity, is you do something and the view changes.
[23:42.400 --> 23:43.860]  So, of course, we can mix and match.
[23:43.960 --> 23:46.420]  We can have dynamic views and side-by-side views.
[23:46.620 --> 23:47.900]  This is an example of that.
[23:48.320 --> 23:49.300]  What else are we doing?
[23:49.480 --> 23:51.640]  Well, there's a lot of different approaches to navigation.
[23:51.980 --> 23:56.120]  I've already talked briefly about this idea of geometric zooming versus semantic zooming.
[23:56.800 --> 23:58.960]  A lot of systems allow things like panning.
[23:59.760 --> 24:02.720]  And then I'll talk a bit more about constrained navigation in a minute.
[24:04.120 --> 24:05.900]  Now, I mentioned superimposing layers.
[24:06.340 --> 24:07.520]  That's a design choice.
[24:07.840 --> 24:13.080]  So, in particular, when you're going to chop things up into pieces, you could put those pieces side-by-side.
[24:13.080 --> 24:14.460]  That's the juxtaposed views.
[24:14.680 --> 24:17.660]  Or you could actually combine layers on top of each other.
[24:17.800 --> 24:18.920]  That's superimposing.
[24:19.320 --> 24:23.160]  And once again, design space, there's a limit as to how many things you can superimpose.
[24:23.500 --> 24:25.300]  Line charts are one of the best cases.
[24:25.300 --> 24:27.300]  You might be able to get up to a dozen or so.
[24:27.420 --> 24:32.780]  If you've got a more complex thing that you're superimposing often, you might just have a foreground and a background layer.
[24:32.980 --> 24:34.720]  These might be dynamically determined.
[24:35.320 --> 24:40.920]  But it's useful to think about this as a design choice in the same context as juxtaposing things side-by-side.
[24:43.520 --> 24:49.520]  Now, one of the key things is as your data set gets big enough, at some point you have to give up on this idea that you show the whole thing.
[24:49.700 --> 24:50.380]  You can't.
[24:50.440 --> 24:51.140]  It's too big.
[24:51.140 --> 24:59.780]  You're going to have some form of either static or often dynamic reduction of what you show at any given instant in any given view.
[25:00.500 --> 25:04.100]  And the two classic examples for reducing is one is filtering.
[25:04.100 --> 25:05.600]  You can throw things away.
[25:06.120 --> 25:08.260]  And the other is aggregation.
[25:08.440 --> 25:17.060]  You don't draw everything, but the things that you've aggregated away, you've got some stand-in proxy where you're trying to show the important aspects of that in the view.
[25:17.960 --> 25:23.820]  And so a lot of visualization comes down to interesting complex combinations of filtering and aggregation.
[25:24.220 --> 25:34.980]  One whole set of families of idioms has to do with trying to combine filtering and aggregation together where you've got focus and contextual information in one view.
[25:35.220 --> 25:37.660]  This is often called just the buzzword focus plus context.
[25:37.960 --> 25:39.260]  Here I'm calling it embedding.
[25:39.360 --> 25:47.820]  And this sort of stretch-and-squish navigation that you saw is an example of that combination of filtering and aggregation together in one view.
[25:48.020 --> 25:52.620]  In particular, this is in the class of things where geometry is actually distorted.
[25:52.800 --> 25:54.560]  You saw how you could stretch things out.
[25:54.840 --> 25:57.020]  The particular metaphor of stretching and squishing.
[25:57.140 --> 25:58.760]  There's others like using a fisheye lens.
[25:59.900 --> 26:03.260]  And so this is actually an example of constrained navigation.
[26:03.620 --> 26:06.880]  The idea is to make it the case that nothing can be off-screen.
[26:07.240 --> 26:13.260]  Things can be compressed arbitrarily small at the borders, but you don't actually have to worry what's behind my head or outside my field of view.
[26:13.260 --> 26:15.840]  So it's an example of a constraint on navigation.
[26:16.860 --> 26:19.200]  In particular, we can classify this more.
[26:19.300 --> 26:21.180]  The shape is rectilinear as opposed to radial.
[26:21.500 --> 26:24.520]  There's actually multiple foci because you can do this many times.
[26:25.040 --> 26:30.360]  And everything changes in the whole view as opposed to some kind of transformation that has purely local effect.
[26:33.140 --> 26:33.860]  What else?
[26:34.060 --> 26:34.700]  There's more.
[26:34.960 --> 26:39.480]  So there was a part of the video where we were actually changing the spatial arrangement of the views.
[26:39.480 --> 26:41.420]  Actually, I'm trying to think if that was in the clip I showed.
[26:42.280 --> 26:43.700]  That might have been in the later parts.
[26:43.820 --> 26:46.920]  But a crucial idea is reordering the view.
[26:47.120 --> 26:50.880]  This is a matrix where you could actually pick which column of data to order by.
[26:51.700 --> 26:53.580]  You can also reorder items as you need.
[26:54.300 --> 26:55.960]  So why do we do this?
[26:56.120 --> 26:58.820]  Well, what that lets you do is check for correlations.
[26:58.820 --> 27:05.700]  Because if you reorder by one column, you can see whether things are still in a similar pattern or a different pattern across the other columns.
[27:05.700 --> 27:11.320]  So there's one example that comes on later on in the video where we sort by CPU usage.
[27:11.500 --> 27:15.780]  And then we look and say, ah, there's a bunch of examples of high load, but there's not high CPU.
[27:16.000 --> 27:18.940]  So, well, maybe that's an example where it's an IO bound process.
[27:19.380 --> 27:22.160]  That's the kind of thing we can then figure out.
[27:22.340 --> 27:23.700]  So what are we doing here?
[27:23.840 --> 27:26.700]  This is another example of changing the view over time.
[27:26.860 --> 27:30.840]  But now instead of just thinking about this as navigation, you're thinking about this as resorting.
[27:31.700 --> 27:35.460]  Pretty much everything that involves changing a view is going to involve some kind of selection.
[27:35.780 --> 27:37.460]  So that's another thing we want to think about explicitly.
[27:37.900 --> 27:42.400]  There's a lot of things about what are the semantics of selection that I'm not going to get into in detail today.
[27:44.880 --> 27:49.040]  But a crucial idea is, let's get to this question of visually encoding.
[27:49.260 --> 27:51.520]  I've been talking about a lot of the interaction approaches.
[27:51.840 --> 27:55.780]  Let's think about this question of how are we even drawing things on the screen in the first place.
[27:55.780 --> 28:00.160]  That there's a lot of questions about how do we arrange this data in space.
[28:00.840 --> 28:05.240]  And here I'm talking in particular about the order that I'm drawing things on.
[28:05.780 --> 28:08.200]  And in fact, it's not even just ordered, it's also aligned.
[28:08.360 --> 28:09.460]  I have this matrix format.
[28:10.300 --> 28:11.980]  So, what's that all about?
[28:13.860 --> 28:20.000]  Now, in order to actually talk coherently about these options for arranging space, I'm going to introduce two definitions.
[28:20.480 --> 28:22.120]  One is this idea of a mark.
[28:22.120 --> 28:24.220]  It's some geometric primitive you're using.
[28:24.380 --> 28:24.840]  It's a point.
[28:25.020 --> 28:25.460]  It's a line.
[28:25.600 --> 28:26.080]  It's an area.
[28:26.740 --> 28:29.660]  So, this is a primitive that represents a data point.
[28:29.940 --> 28:32.300]  And in particular, what can you do with that?
[28:32.480 --> 28:35.680]  You can encode information using these things I'm going to call visual channels.
[28:36.360 --> 28:39.860]  So, those control the appearance of these geometric marks.
[28:40.020 --> 28:40.840]  And what are they?
[28:41.060 --> 28:43.740]  Well, it's things like spatial position on the screen.
[28:43.980 --> 28:46.200]  Or if you've got a point, you can size code it.
[28:46.340 --> 28:48.620]  You can color code these points and lines and areas.
[28:48.760 --> 28:50.620]  You can orientation code with an angle.
[28:50.620 --> 28:54.940]  You can have things like size coding, whether it's the length of a line or the area of a box.
[28:56.840 --> 28:58.860]  So, this is a lot of possibilities.
[28:58.860 --> 29:00.080]  How do we pick?
[29:01.700 --> 29:13.440]  This is where we get at one of the crucial issues about matching up perception with these design choices, which are that there's two crucial different kinds of these visual channels.
[29:13.440 --> 29:20.480]  One set of channels, I'm going to call these the magnitude channels, is a good match for data that intrinsically has an ordering.
[29:20.700 --> 29:24.600]  Because your brain interprets this as a how much kind of a thing.
[29:24.600 --> 29:34.840]  So, you can actually express magnitudes directly with these channels like position, particularly position along a common scale where you've got alignment.
[29:35.380 --> 29:38.100]  Things like orientation can express magnitudes.
[29:38.420 --> 29:43.620]  Area, the amount of black and white or the amount of saturation in a color.
[29:43.620 --> 29:45.690]  So, these are all ways of expressing magnitudes.
[29:46.340 --> 29:48.920]  And a crucial thing is some are better than others.
[29:49.240 --> 29:55.560]  And by better, I mean more accurate in terms of your ability to perceive them.
[29:55.800 --> 29:59.040]  There's a lot of other attributes as well that I'm not going to have time to talk about today.
[29:59.860 --> 30:10.200]  But I want to distinguish between these magnitude channels for ordered data and these other identity and location channels that are well suited for the categorical kinds of data.
[30:10.200 --> 30:12.520]  So, what region of space is it in?
[30:12.600 --> 30:13.640]  What color is it?
[30:14.220 --> 30:18.340]  You know, what is the moving stuff versus the non-moving stuff?
[30:18.480 --> 30:19.760]  What's the shape of a point?
[30:20.140 --> 30:22.140]  So, this is a crucial divide.
[30:22.300 --> 30:27.580]  And once again, these identity channels also are listed according to effectiveness.
[30:28.420 --> 30:30.320]  So, there's a crucial thing.
[30:30.640 --> 30:33.180]  Space is at the top, on both sides.
[30:33.180 --> 30:44.380]  So, whether your data is order or categorical, spatial position is really important in terms of shaping how it is that you perceive and thus how you think about this data.
[30:45.660 --> 30:48.000]  So, there's actually a lot more to say about this.
[30:48.100 --> 30:51.700]  This could be an hour-long talk in and of itself about why is this true?
[30:51.840 --> 30:53.440]  Why are these rankings are the way they are?
[30:53.640 --> 30:54.840]  What are their implications?
[30:55.760 --> 30:57.200]  Don't have time today, alas.
[30:57.360 --> 31:00.700]  But if you're curious, there's a talk there that goes into that in more detail.
[31:01.720 --> 31:06.780]  What I'm going to talk about is the implications of these for visualization design.
[31:07.360 --> 31:09.580]  And I said I couldn't quite keep away from algorithms.
[31:09.780 --> 31:11.340]  So, I'll just give a brief mention to them.
[31:12.340 --> 31:15.020]  For this particular system, it was actually a front-end back-end.
[31:15.160 --> 31:17.900]  Back-end was a big honking database from AT&T called Swift.
[31:18.500 --> 31:24.000]  On the front-end was a whole rendering framework we ended up building that's documented in this other paper on Prysad.
[31:24.240 --> 31:29.520]  We had separate threads, as we mentioned in the video, for rendering versus server update to always give interactive response.
[31:30.280 --> 31:34.020]  It's worth saying there are interesting graphics and algorithmic issues here.
[31:34.460 --> 31:37.200]  We had this idea that marks might mean something.
[31:37.780 --> 31:42.440]  And so we want to make sure these marks always show up, even if you squish things down to very small.
[31:42.560 --> 31:47.460]  Once you've got millions of items, you might have a very small number of pixels in which to express a lot of these items.
[31:47.700 --> 31:54.400]  If you just throw this at a standard graphics card, it's going to actually call those away, and the things that are semantically meaningful, you're not going to see.
[31:54.580 --> 32:00.260]  So we thought a lot about how to do a software rendering infrastructure that actually doesn't depend on the number of items.
[32:00.660 --> 32:08.240]  We want the rendering to be based on the number of pixels on screen, which is constant, as opposed to the size of the data set, which can grow more and more.
[32:09.340 --> 32:11.380]  And we actually had a generic framework for doing this.
[32:11.460 --> 32:13.020]  It's not just for time series data.
[32:13.020 --> 32:15.840]  We showed it with gene sequences and very large trees.
[32:16.040 --> 32:19.140]  But I'm not going to get too deep into that today, even though it's a fun topic.
[32:19.140 --> 32:27.100]  I just want to make sure that you understand that there's some connections between what you've specified at your idiom level and then some algorithmic challenges in actually building that.
[32:29.120 --> 32:29.720]  All right.
[32:30.180 --> 32:30.780]  Marching along.
[32:31.140 --> 32:34.560]  So that was brief, but it gave you some sense of starting in on the design space.
[32:34.720 --> 32:38.320]  And I'm going to have a second system to keep marching us through this design space.
[32:38.440 --> 32:39.640]  This one's called Overview.
[32:40.160 --> 32:43.320]  And this was done for basically document mining for journalists.
[32:44.720 --> 32:52.400]  So this was joint work with Matt Bramer and Steven Ingram from my own lab and Jonathan Stray, who was Associated Press at the time that I met.
[32:54.200 --> 32:56.740]  So for a movie metaphor, here's the origin story.
[32:56.980 --> 32:58.560]  So this is WikiLeaks meets Glimmer.
[32:59.220 --> 33:01.120]  So let me say what I mean by that.
[33:01.940 --> 33:03.760]  Jonathan Stray is a hacker journalist.
[33:03.760 --> 33:06.560]  I met at another hacker conference about four years ago.
[33:06.820 --> 33:09.300]  And he was analyzing the Iraq war logs.
[33:09.300 --> 33:20.520]  And he had this conjecture that in this data set, that there was a classification that was actually intrinsic, where they labeled everything by a small set of labels, things like friendly action versus criminal incident.
[33:20.720 --> 33:29.820]  And his conjecture was that there was more structure than just this very high-level categorization in the document data set, and he wanted to try to discover it to actually report on it.
[33:29.940 --> 33:38.600]  And he had already done quite a bit of natural language processing on the data set, but he needed better viz tools to try to understand what some of the smaller-scale structure was.
[33:39.780 --> 33:54.340]  We had published a paper a few years ago that was a very algorithmic paper about dimensionality reduction, and we were very happy that we were using a multi-level algorithm to really get traction where we could get tens of thousands of documents and terms and get reasonable performance.
[33:54.780 --> 33:58.020]  So Jonathan noticed this in the literature and said, great, I'll use this.
[33:58.340 --> 34:01.480]  And even still, things fell short.
[34:01.540 --> 34:04.460]  And we thought, okay, great, we'll do a project together, and we'll build the system.
[34:05.620 --> 34:13.980]  So let me talk about what we thought we were building at the beginning, because we did have a pretty clear understanding, at least of the task that Jonathan was trying to do.
[34:15.440 --> 34:24.740]  So this is where the story starts getting complex, and it's interesting to think about the data abstraction, because if I say I've got 10,000 documents, well, what do you do?
[34:24.960 --> 34:25.920]  You can read them all.
[34:26.080 --> 34:26.580]  That's reading.
[34:26.800 --> 34:28.300]  What if you want to visualize them?
[34:28.300 --> 34:30.660]  That's not directly a form that you can visualize.
[34:30.900 --> 34:34.900]  You have to do some kind of derivation of new data from the raw data.
[34:35.100 --> 34:37.480]  And this is not just about this particular problem.
[34:37.640 --> 34:46.540]  This is, in general, there's a lot of the interesting action in visualization comes from transforming the data from how you got it into what the user actually needs.
[34:46.920 --> 34:51.120]  So in particular, this case of deriving, what do you do with text?
[34:51.360 --> 34:56.120]  Well, a very common thing is you go from a bunch of documents to a very, very, very high dimensional table.
[34:56.120 --> 34:58.960]  So we're seeing this table data set again.
[34:59.880 --> 35:00.700]  What do we do?
[35:00.920 --> 35:04.080]  Well, this is an old model, an information retrieval, the so-called bag of words model.
[35:04.420 --> 35:12.300]  What you do is you say that these attributes in your giant matrix is any word that appears anywhere in the whole collection is a column in your table.
[35:12.600 --> 35:18.100]  And then for each item, which is a document, you say how many times does that word appear in the document.
[35:18.280 --> 35:23.500]  And so what you get is a very sparse high dimensional table because a lot of documents won't have particular words in them.
[35:24.820 --> 35:28.100]  So you don't just say, all right, I've got a high dimensional table.
[35:28.240 --> 35:29.500]  I'm going to directly visualize that.
[35:29.640 --> 35:30.620]  You do another derivation.
[35:31.240 --> 35:35.600]  Your next step is let's go from text to high dimensional table to a low dimensional table.
[35:36.520 --> 35:50.760]  So what you want to do, and this is where it gets interesting and tricky from the math, is you want to synthesize a small number of new dimensions that capture most of the proximity relationships in this high dimensional feature vector space that I've just described.
[35:51.900 --> 35:52.920]  Well, then what?
[35:53.100 --> 35:58.380]  Then what you can do is you can actually plot things in the low dimensional space and look for clusters of items.
[35:58.540 --> 36:00.020]  And that's the interesting part.
[36:00.400 --> 36:10.280]  So two derivation steps where we actually need to go from text through high dimensional table to low dimensional table and then to look at clusters.
[36:10.500 --> 36:17.040]  And these are the things we're trying to see is clusters of documents that might have some intrinsic connection of meaning.
[36:17.240 --> 36:26.540]  So now we can finally go from producing this derived data to this idea that we're actually doing a discovery task where we're consuming the data that we've just produced.
[36:26.740 --> 36:33.360]  And often you... the two flavors of discovery are usually you're generating a new hypothesis or you're verifying an existing one.
[36:33.860 --> 36:35.720]  And these were both of interest here.
[36:38.180 --> 36:39.900]  So that was kind of a lot of words.
[36:40.040 --> 36:45.480]  Let me actually show you this in pictures where we're trying to do actually this linked series of tasks.
[36:45.760 --> 36:49.140]  So we're trying to go from a high dimensional table down to a low dimensional one.
[36:49.500 --> 36:55.540]  Then we take this low dimensional table and we plot these items, which are the documents, just as a scatter plot.
[36:55.960 --> 37:00.940]  And then the question is, do we either see cluster structure or verify conjectured cluster structure?
[37:01.100 --> 37:07.800]  And then at the end, if you actually maybe spot check some of the documents, you look at them and you say, ah, this is the meaning of that cluster.
[37:07.960 --> 37:10.200]  In this case, I've argued that they're all about wombats.
[37:10.200 --> 37:14.360]  Then we can say, okay, we've verified that this cluster exists and has a label.
[37:14.700 --> 37:17.740]  Now, there's a bunch of really interesting math behind this.
[37:17.980 --> 37:22.240]  And a lot of other questions as well about how you design these kind of dimensionality reduction systems.
[37:22.480 --> 37:23.840]  That's also a whole other talk.
[37:24.360 --> 37:26.760]  And I'm not going to go deeper into that today.
[37:28.680 --> 37:33.180]  What I am going to do is show you the overview version one video first.
[37:33.180 --> 37:34.620]  Again, just a snippet.
[37:45.940 --> 37:46.860]  The structure.
[37:48.500 --> 37:52.160]  The last tag set from this journalist covers disparate issues.
[37:52.760 --> 37:56.600]  We expand the disco tree to show smaller clusters as well.
[37:57.440 --> 38:07.040]  The journalist found issues he was not previously aware of, such as this cable which alleged Iranian plans to ship unmanned aerial vehicles to Venezuela.
[38:08.900 --> 38:14.500]  We now switch to another WikiLeaks data set, the Afghan war logs from July 2009.
[38:15.560 --> 38:19.400]  We use modisco tag to quickly tag these 3000 documents.
[38:20.260 --> 38:26.800]  We are certain that the first branch is about found IEDs when we look at the node labels in the active set list.
[38:26.800 --> 38:30.620]  We create the tag and add this node to it.
[38:31.900 --> 38:34.620]  The next branch appears to be about engagements.
[38:35.140 --> 38:42.940]  The abbreviations in these reports are INS for insurgents, COI for company and SAF for small arms fire.
[38:44.100 --> 38:46.560]  The next branch is about fire missions.
[38:47.160 --> 38:54.000]  These branches form very tight clusters in the MDS view compared to the more diffuse layouts of the previous data set.
[38:55.360 --> 39:00.420]  This group has labels with SALTER, the acronym for the After Action Report format.
[39:01.680 --> 39:07.780]  We select a visible cluster in the MDS view and see that it corresponds to a branch in the disco tree.
[39:08.340 --> 39:12.200]  We tag this group of selected documents as detainee transfers.
[39:13.260 --> 39:26.060]  The combination of the disco tree, tagging, and a traditional MDS view supports machine-assisted human categorization to capture the concepts of semantic interest in document collections.
[39:26.380 --> 39:27.580]  Peer in the act of...
[39:27.580 --> 39:29.200]  Okay, so I'm going to move on.
[39:29.340 --> 39:30.920]  We've just shown a snippet of that video.
[39:35.480 --> 39:39.080]  And again, talk about these questions of why and what and how.
[39:39.400 --> 39:41.220]  So you might have noticed some things.
[39:41.380 --> 39:43.860]  I wasn't just showing a scatter plot.
[39:44.080 --> 39:45.060]  What was that all about?
[39:45.060 --> 39:50.860]  So once we had thought about, well, why are people looking at these clusters in these scatter plots?
[39:51.000 --> 39:56.800]  If the job is to understand the cluster structure, are there other kinds of support we can offer for this task?
[39:57.000 --> 40:04.960]  What we ended up doing was deriving yet more data, a third step, of the entire hierarchy of possible clusterings.
[40:05.240 --> 40:08.180]  And we offered them a direct view of that.
[40:08.180 --> 40:14.120]  So what are we doing by letting them support this space of possible cluster hierarchies of these documents?
[40:14.460 --> 40:16.620]  Well, now we've moved into another data set type.
[40:16.860 --> 40:23.920]  Now instead of thinking just about tables where you've got items with attributes, now let's think where these items might actually have links between them.
[40:24.300 --> 40:30.280]  So now with network structures, often we call these items nodes, and we think explicitly about the links between them.
[40:30.680 --> 40:32.380]  Trees are special cases of networks.
[40:32.600 --> 40:36.780]  So now we're thinking about the cluster hierarchy that's a tree kind of a data set.
[40:38.220 --> 40:40.900]  When you're dealing with trees, what's your task?
[40:41.080 --> 40:48.180]  Well, abstractly, it almost always comes down to understanding some aspect of topological structure or path tracing through these networks.
[40:49.540 --> 40:51.540]  So how can you show cluster hierarchies?
[40:51.760 --> 40:54.080]  Well, again, lots of approaches.
[40:54.080 --> 40:59.220]  A lot of them fall under the general heading of you're explicitly drawing nodes connected by links.
[40:59.360 --> 41:02.180]  So there's marks that represent links, there's marks that represent nodes.
[41:02.460 --> 41:05.500]  So these so-called node link diagrams are a really common way of doing that.
[41:05.500 --> 41:08.520]  And that's what we were doing in that view on the upper left in the video.
[41:09.200 --> 41:15.120]  There was also this idea of supporting tagging of either clusters or documents more specifically.
[41:15.400 --> 41:20.420]  And that's an example of another kind of production of data along the way in a visualization.
[41:20.520 --> 41:24.160]  Where the goal of the user actually is to add information.
[41:24.160 --> 41:28.220]  In this case, annotating clusters with semantically meaningful tags.
[41:28.920 --> 41:35.960]  What's interesting about that is the tagging can either follow the hierarchy that you've come up with, or completely cross-cut it.
[41:36.480 --> 41:39.400]  There's some interesting work that shows there's a huge number of possible clusterings.
[41:40.200 --> 41:48.720]  And so instead of thinking we'll find the perfect one automatically, which might be a very quixotic quest, then we allow tagging in these ways that could cross-cut or follow it.
[41:48.720 --> 41:58.120]  So it allows you not only to do this kind of annotation, but also there's some cases where you might actually want to read either some or all of the documents, not just spot-check a few.
[41:58.360 --> 42:00.340]  You can keep track of how far you've gotten in that.
[42:00.480 --> 42:07.540]  And any kind of semantics that may or may not match the cluster structure is something you can explicitly do then with tagging.
[42:08.720 --> 42:15.820]  So going back to design decisions, that same idea of multiple linked views juxtaposed side by side, we're doing that again.
[42:16.040 --> 42:17.500]  This is a very, very common idiom.
[42:17.940 --> 42:20.200]  I mentioned this idea of linking between views.
[42:20.380 --> 42:31.620]  In this case, what we're doing is we're color coding where we've got the dots in the scatter plot, we've got the branches in the tree, we actually have the tags themselves, and we show that they're all linked by having them in the same color.
[42:31.880 --> 42:33.120]  So that sounds straightforward.
[42:33.940 --> 42:37.280]  There's actually a lot of interesting and complex ways to use color in visualization.
[42:37.280 --> 42:47.800]  Again, no time to get into it today, but suffice it to say that for categorical data, then picking a small number of distinguishable colors is a very effective way to show those distinctions.
[42:50.060 --> 42:54.660]  Now, so we had also some specific views for reading both the clusters and the documents.
[42:56.340 --> 42:57.720]  So where did this all end up?
[42:57.820 --> 43:00.880]  At the algorithm level, we'd gotten deep into the algorithm.
[43:01.060 --> 43:05.320]  We thought a lot about how do you construct hierarchies quickly for these clusters when your data is very sparse.
[43:05.320 --> 43:07.500]  We went off, we built a research prototype.
[43:07.900 --> 43:13.380]  We were really happy that we got the Associated Press Bureau Chief of Caracas to actually investigate that WikiLeaks data set.
[43:13.520 --> 43:14.760]  I showed a bit of that in the video.
[43:16.120 --> 43:20.980]  And it did become clear that if we really want a journalist to use this, well, there were going to be some barriers.
[43:22.340 --> 43:23.480]  Installation is difficult.
[43:23.840 --> 43:25.220]  Loading in their data was difficult.
[43:25.660 --> 43:30.880]  So this is often where a lot of research papers end is, you know, here is our prototype, got some promising initial results.
[43:31.120 --> 43:37.000]  What was great about in this case is we got money from the Knight Foundation to actually build something real for journalists.
[43:37.620 --> 43:48.920]  So where we could actually have it be web deployed and integrate with a system called Document Cloud that a lot of journalists use, and actually worry about questions like usability, because what do you know, research prototypes are often not the most usable thing on the planet.
[43:49.320 --> 43:54.480]  So many months of engineering went into actually making something that people could use.
[43:54.620 --> 44:02.800]  And the success story here was that a reporter did use the system, who had no connection to us, in this case to investigate police corruption in Tulsa.
[44:02.960 --> 44:04.400]  So we were very excited about this.
[44:05.580 --> 44:10.040]  But speaking of very excited, we started out being very excited about dimensionality reduction.
[44:10.360 --> 44:21.300]  And even though, in the end, we decided that was a great problem in its own right, we didn't need to go deep into it for that particular system, but we went off and did it, thinking a lot about this question of sparse data.
[44:21.500 --> 44:31.020]  So just for those who are curious, we now have a system that does scale to millions of documents, and that's going to appear in a machine learning journal quite soon, and the preprints posted if you want to know more about that algorithm.
[44:32.700 --> 44:37.020]  But back to this problem, it turned out we weren't there yet.
[44:37.400 --> 44:43.100]  It required a few more years of work to actually get to the point of really understanding what the users needed.
[44:43.400 --> 44:55.260]  So this question of which views do they need, and what exactly should they show, and how precisely in terms of the idiom design should they show it, that's a question not just of usability, but also deep questions of utility.
[44:55.460 --> 44:57.060]  What do they need to do that?
[44:57.880 --> 45:01.740]  And it turned out that along the way there was actually two more versions.
[45:02.300 --> 45:03.820]  Then people really did start adopting.
[45:04.020 --> 45:10.400]  So there was a story about Ryan being hypocritical when he said at the campaign trail, cut this, and then as a congressman he said, hey, give me that.
[45:10.880 --> 45:12.720]  A story about the gun control debate.
[45:13.040 --> 45:18.280]  A follow-up about whether or not there were more stories merited in a data set about corruption in Texas.
[45:18.280 --> 45:29.080]  We were particularly excited that there was a story about police corruption in New York, in Newsday, that was nominated for the Pulitzer Prize as a finalist, and that actually used overview for the investigation.
[45:29.560 --> 45:33.800]  So this was a long pass to actually get journalists to adopt it.
[45:34.020 --> 45:36.560]  What's interesting is, let me show you the final video.
[45:36.700 --> 45:39.200]  It doesn't look so much like the original.
[45:39.520 --> 45:40.940]  Again, I'm just going to show a snippet.
[45:46.280 --> 45:50.260]  You begin your analysis at the root folder, which contains all documents.
[45:51.080 --> 45:56.280]  Each folder is labeled by the key words from the documents filed in that folder.
[45:57.040 --> 46:02.460]  Overview tells you if most, some, or all documents contain each word or phrase.
[46:02.820 --> 46:10.280]  In this case, we have a folder where all documents contain the word letter and the phrases President Obama and gas exploration.
[46:10.520 --> 46:12.120]  So let's look at some of these documents.
[46:13.680 --> 46:16.900]  I can flip through them quickly using the next and previous buttons.
[46:17.640 --> 46:19.560]  And it looks like they're all of the same.
[46:19.860 --> 46:20.880]  It's a form letter.
[46:21.400 --> 46:29.900]  What happened here is 66 congresspeople wrote a form letter to President Obama, and President Obama wrote 66 form letters back.
[46:30.880 --> 46:33.860]  We can tag this folder to keep track of what we've learned.
[46:34.600 --> 46:36.240]  Let's call it form letter.
[46:37.520 --> 46:38.040]  There.
[46:38.240 --> 46:39.340]  Now it appears in the tree.
[46:41.140 --> 46:43.400]  This folder is a little less clear.
[46:44.440 --> 46:46.840]  No word appears in every document.
[46:47.280 --> 46:48.600]  So let's open it up.
[46:48.940 --> 46:53.120]  Still, we're looking at a folder which probably contains a bunch of different topics.
[46:53.320 --> 46:54.340]  So let's open it up again.
[46:55.740 --> 47:01.500]  Ah, now we've got a folder where every document has the word investigation and Atlantis platform.
[47:02.800 --> 47:12.560]  What this is, is an email thread about a citizens group which was asking for an investigation into the safety of a particular oil drilling platform called the Atlantis.
[47:12.880 --> 47:14.380]  So let's tag that too.
[47:19.200 --> 47:25.540]  Next to it are a bunch of emails about legislators' trips to a particular oil drilling platform.
[47:25.540 --> 47:27.020]  We can tag that too.
[47:27.380 --> 47:28.660]  We'll call it revisit.
[47:31.580 --> 47:37.120]  Notice if I create a tag while I'm looking at a particular document, only that one document gets tagged.
[47:37.320 --> 47:41.860]  If I want to tag all of it, I can go back to the list and hit this plus button.
[47:42.480 --> 47:49.780]  Using the plus and minus buttons, I can combine tags or subtract them, and each document can have any number of tags.
[47:49.780 --> 47:53.340]  If we're looking for something in particular, we can search for it.
[47:54.360 --> 47:58.380]  Let's try environmental impact.
[48:00.020 --> 48:03.820]  Overview shows us where in the folders those documents appear.
[48:04.180 --> 48:08.140]  If we want to look a little more closely, we can turn that search into a tag.
[48:09.160 --> 48:09.680]  There.
[48:09.800 --> 48:15.480]  Now we can drill down into the folders to find exactly what these documents concern.
[48:15.740 --> 48:18.660]  Here's a bunch of documents talking about marine rambles.
[48:18.660 --> 48:22.160]  All right, I'm going to stop now and go back.
[48:22.500 --> 48:27.040]  And what's interesting about this is there's a lot of differences from where we started.
[48:27.200 --> 48:30.620]  For one thing, we don't even show that dimensionality reduction scatterplot anymore.
[48:30.660 --> 48:33.660]  We found for this particular kind of exploration, they didn't need it.
[48:34.000 --> 48:35.500]  The tree looks quite different.
[48:35.640 --> 48:41.740]  The emphasis is on the nodes so that you can think what's the structure of this cluster much more than the topological structure of the links.
[48:42.000 --> 48:44.540]  Other things like the document and the cluster viewer got combined.
[48:44.540 --> 48:49.920]  So, the interesting question is why were people really using this?
[48:50.020 --> 48:56.560]  We couldn't figure out until we built it and deployed it and figured a lot of stuff out along the way about this idea of what's the task.
[48:56.900 --> 48:59.680]  We started out with this idea that the task was what's in this collection.
[48:59.880 --> 49:02.020]  In part because the use case was leaked documents.
[49:02.020 --> 49:06.500]  And that's things like generating hypothesis, summarizing clusters, exploring them.
[49:06.720 --> 49:09.580]  And then we found people used it for these interestingly different things.
[49:09.680 --> 49:18.000]  When you're locating evidence within something like a Freedom of Information Act dump, then it's often things like verify a hypothesis rather than start from scratch.
[49:18.540 --> 49:20.160]  Identify the things of interest to that.
[49:20.480 --> 49:22.440]  Locate them within the larger data set.
[49:22.840 --> 49:27.700]  So, thinking about this systematically, well, generate and verify were examples of this discovery task.
[49:27.700 --> 49:34.720]  We now see some different flavors of identify one thing versus summarize them all in terms of the level of these queries.
[49:35.200 --> 49:36.760]  But what about explore and locate?
[49:37.340 --> 49:40.480]  We actually found that it was useful for us to think more precisely.
[49:40.720 --> 49:42.560]  We've just been tossing around this word explore.
[49:42.860 --> 49:44.100]  A lot of his papers do that.
[49:44.500 --> 49:46.060]  We wanted to think of it more precisely.
[49:46.640 --> 49:51.460]  Well, think about it whether you know the target and the location.
[49:51.660 --> 49:53.620]  So, if you know the target and location, you're looking it up.
[49:53.800 --> 49:57.200]  If you don't know the target but you have some sense of where it might be, well, then you're browsing.
[49:57.620 --> 50:01.320]  If you know what you're looking for but you don't know where it is, well, then you have to locate.
[50:01.500 --> 50:09.500]  And then we save the word explore for this really unconstrained task of you don't know where it is and you don't know what exactly you're looking for and you want to see what's there.
[50:09.600 --> 50:16.860]  And we found that trying to understand what was going on in this case actually motivated a whole paper in part on these typologies of tasks.
[50:17.560 --> 50:23.420]  It's even harder if you have to prove not here's the smoking gun, but there is no smoking gun in this data set.
[50:23.520 --> 50:24.960]  Prove the absence of evidence.
[50:24.960 --> 50:28.340]  Often you have to exhaustively read every document.
[50:28.640 --> 50:31.640]  We'd thought of this as a way to accelerate people so you don't have to read everything.
[50:31.820 --> 50:36.520]  They ended up using it as support for if they did have to read everything, to even keep track of what they've done.
[50:38.260 --> 50:39.500]  So, people continue.
[50:39.720 --> 50:42.740]  There's some stories on food stamps and credit card repossessions.
[50:42.860 --> 50:44.380]  Just this week, a story on Louis C.K.
[50:44.480 --> 50:45.680]  email a couple of days ago.
[50:45.880 --> 50:52.840]  The Knight Foundation has continued and they're going to fund version 5 for things like naming entities and extracting them and even a plug-in API.
[50:52.840 --> 50:54.200]  So, this is actually live.
[50:54.340 --> 50:55.260]  You can go ahead and play with it.
[50:55.360 --> 50:56.220]  Here's some URLs here.
[50:56.540 --> 50:57.720]  They even have some stickers up front.
[50:57.920 --> 51:00.580]  And I have a five-minute mark, so I will be wrapping up.
[51:01.620 --> 51:02.860]  So, the big picture.
[51:03.020 --> 51:04.800]  I walked you through two specific systems.
[51:05.100 --> 51:07.800]  And this is a small part, actually, of the design space.
[51:07.840 --> 51:10.240]  I spent the last six years thinking about how to describe this.
[51:10.420 --> 51:14.180]  I've got a book that's coming out in a few months that's trying to really walk through everything.
[51:14.420 --> 51:21.860]  And so, what we ended up doing is we walked through a bit of the question of what, a bit of the question of why, a bit of the question of how.
[51:22.320 --> 51:23.540]  There's lots more to think about.
[51:23.740 --> 51:25.580]  Even these questions of channels, as I mentioned.
[51:25.960 --> 51:27.860]  There's a lot more we haven't talked about at all.
[51:28.960 --> 51:31.640]  I'm going to give you one more resource to help you think.
[51:31.840 --> 51:34.280]  I mentioned the idea of these levels of design.
[51:35.380 --> 51:37.960]  And there's two cases that are useful to think about.
[51:38.220 --> 51:40.900]  One is what I'm going to call technique-driven work.
[51:40.900 --> 51:42.300]  You're going to build an algorithm.
[51:42.560 --> 51:46.240]  This is where a lot of us start, especially those of us who are trained as computer scientists.
[51:46.740 --> 51:51.400]  If you want to go the other way and say, here's real users, and they have a problem, and how do I help them?
[51:51.540 --> 51:55.600]  You usually want to start by marching through these levels in the order I've talked about them.
[51:56.140 --> 51:57.420]  These are both great to do.
[51:57.520 --> 51:58.700]  In our group, we do both of them.
[51:58.960 --> 52:02.520]  It's really hard to jump from one to the other if you haven't done it.
[52:02.520 --> 52:03.700]  There's a lot of pitfalls.
[52:03.960 --> 52:09.880]  We actually ended up writing a paper on the methodology of how to do this sort of problem-driven work that we call Design Studies and Viz.
[52:10.140 --> 52:12.220]  This was joint with Michael Siedemeyer and Mariah Meyer.
[52:12.480 --> 52:18.580]  And we actually, between the three of us, we'd done 21 of these over about 10 years in a huge variety of domains.
[52:18.840 --> 52:24.560]  I just picked one to talk about today, the LiveRack one, a newer one that came out as Overview.
[52:24.900 --> 52:31.500]  Two that might be really interesting for you guys, overlay networks in car network engineering, web law analysis.
[52:31.500 --> 52:33.300]  But I'm not going to cover those today.
[52:33.980 --> 52:42.180]  But this paper talks not only about how to do this work in terms of a framework, but 32 specific things you can screw up and how to not screw them up.
[52:42.740 --> 52:45.280]  And many of them learn from some of our own bitter experience.
[52:46.500 --> 52:52.200]  So to wrap up, I analyzed in some detail two systems for server logs and for text.
[52:52.420 --> 52:59.420]  And the big ideas of this analysis were this idea that you should think about your data abstraction, not just to characterize but to derive data.
[52:59.740 --> 53:04.640]  Your task abstraction and how to go from something domain specific to generic in the language of viz.
[53:04.840 --> 53:10.980]  This question of idioms for encoding and interaction in a way before you get to algorithms.
[53:11.460 --> 53:16.900]  And to think about this more systematically with the idea that if you describe what exists, it can help you generate something new.
[53:17.680 --> 53:19.240]  So these talk slides are up.
[53:19.400 --> 53:20.780]  There's lots and lots of other stuff on the website.
[53:20.780 --> 53:22.660]  I mentioned this book that will come out shortly.
[53:23.120 --> 53:24.340]  Thanks to funders.
[53:24.720 --> 53:26.820]  And I think we've got three minutes for questions.
[53:27.280 --> 53:28.100]  Two minutes for questions.
[53:28.860 --> 53:30.100]  Two minutes for questions.
[53:30.880 --> 53:32.260]  I can't see, so yell.
[53:33.020 --> 53:33.460]  Yes?
[53:33.840 --> 53:36.680]  What do you think are some of the best tools for data visualization?
[53:38.160 --> 53:42.700]  If you only have a day to start, I would say go with D3.
[53:43.720 --> 53:45.960]  It's one of the most powerful tools out there.
[53:46.140 --> 53:47.440]  It's gotten widespread traction.
[53:47.440 --> 53:48.900]  At some point, you'll hit a wall.
[53:49.020 --> 53:52.620]  So if you have giant data sets and you need to do complicated interaction, that's not what you want.
[53:52.760 --> 53:57.300]  But if you want to do something simple, web deployable charts, then it's a great place to start.
[53:58.500 --> 54:07.540]  The other question is if you've got interesting data that you want to analyze more in a sandbox way as opposed to deploying tools to someone else, then Tableau is quite a powerful system.
[54:08.500 --> 54:08.880]  Matt?
[54:09.280 --> 54:13.560]  So what's the relationship between the visualization you're doing and machine learning?
[54:13.560 --> 54:16.200]  You mentioned the word machine learning a couple of times.
[54:16.820 --> 54:19.400]  And it looks like this sort of complements that.
[54:19.640 --> 54:24.440]  Yeah, so the question for the mic is how does Viz relate to machine learning?
[54:24.740 --> 54:25.680]  Does it complement it?
[54:25.800 --> 54:28.140]  And the idea is, yeah, it does complement it.
[54:28.320 --> 54:33.820]  There's a place where some of us use some of the same algorithmic techniques, things like dimensionality reduction as an algorithmic place of overlap.
[54:34.020 --> 54:37.500]  But conceptually, you think of them typically as very complementary.
[54:37.500 --> 54:53.200]  That often, in fact, back in this little design study methodology suitable diagram, we talk specifically about if everything you need is in the computer in terms of your data and you completely understand the task, it's a good candidate for machine learning.
[54:53.580 --> 54:59.700]  But if you're not sure what you need yet, that kind of exploratory space is often where Viz comes into play.
[55:00.140 --> 55:06.200]  And pure Viz of just let me draw everything and let the user figure it all out from scratch, well, you can only do that if the data sets small.
[55:06.560 --> 55:15.700]  So this kind of scaffolding where you do some amount of automation and computation and then you show those results visually and you're scaffolding back and forth, that's usually the interesting use case.
[55:17.520 --> 55:19.020]  Maybe one last quick question.
[55:19.820 --> 55:20.560]  Going once.
[55:21.440 --> 55:22.140]  Going twice.
[55:23.500 --> 55:23.900]  All right.
[55:24.080 --> 55:24.740]  Onward to Ellsberg.
[55:25.020 --> 55:25.260]  Thanks.
[55:36.950 --> 55:40.010]  And just so you know, we'll be simulcasting the keynote.
[55:40.830 --> 55:41.590]  Everything's full.
[55:41.870 --> 55:47.390]  So if you want to see it, you're probably better off keeping your butt in the seat here rather than looking...