[00:01.700 --> 00:03.340]  Hey, my name's Saber.
[00:04.720 --> 00:07.940]  Yeah, today I'll be talking about the Arts Open Framework.
[00:08.540 --> 00:11.920]  My name and email are up here and there's also a link to these slides.
[00:12.300 --> 00:16.200]  I'm just hosted on GitHub pages if anyone wants to follow along that way.
[00:17.260 --> 00:21.400]  Make sure I'm on the right page to be clicking here.
[00:23.300 --> 00:26.880]  All right, so first for a little background about me.
[00:28.260 --> 00:30.120]  This is my second time at HOPE.
[00:30.120 --> 00:31.900]  I really enjoyed last year.
[00:32.060 --> 00:34.440]  I got my ham radio license, which is super fun.
[00:36.000 --> 00:39.120]  Yeah, I mostly am an engineer.
[00:39.300 --> 00:40.700]  I build tools for researchers.
[00:41.240 --> 00:48.480]  If you were at my workshop on Friday, we looked at one of those tools, Licorice, and did some real-time audio processing.
[00:49.000 --> 00:51.660]  I also build tools for automation.
[00:52.200 --> 00:58.360]  I do a lot of hybrid provisioning of workstations and computers used to self-host.
[00:58.920 --> 01:02.620]  And I'm an electrical engineer by trade, so I love to work with batteries.
[01:02.620 --> 01:06.120]  I love to work with lighting, audio, you name it.
[01:06.260 --> 01:09.300]  And this is all kind of coming from a background in systems neuroscience.
[01:10.620 --> 01:11.100]  Cool.
[01:11.480 --> 01:19.620]  So today we're going to talk about kind of a problem that we've been dealing with in the research space for a little while.
[01:20.420 --> 01:30.020]  And this is, I guess, that we have what's a standard research output, the paper, and you have paper with figures, tables.
[01:31.180 --> 01:36.120]  And more recently, people will be including data and code for those papers.
[01:36.920 --> 01:41.360]  And oftentimes you can find the data, the data is linked to the paper, but how do you actually use it?
[01:41.740 --> 01:46.140]  It's been kind of a hard thing as a researcher and someone who just is interested in science.
[01:46.780 --> 01:52.780]  You know, I see the results, but if I actually want to dive into them, if I want to build off of them, collaborate, it's hard to do that.
[01:53.580 --> 01:58.100]  A big part of this is people in research work in a lot of different development environments.
[01:59.040 --> 02:04.300]  They're using different hardware and might have a really complicated set up to collect their data.
[02:04.780 --> 02:09.560]  But more and more at the end of the day, people are using the same kind of analysis tools.
[02:09.780 --> 02:12.480]  So people are running code on their data.
[02:12.740 --> 02:21.480]  They're often using like a cloud provider or a local cluster to do their analysis and they generate figures and results that way and include them in their output.
[02:22.700 --> 02:33.340]  So one goal we had and sort of this problem we've seen over the years, it's hard to run code that even, you know, even if I wrote code a year ago, I might have trouble running my own code.
[02:33.540 --> 02:37.400]  And so we definitely want that to be easy, but we also want other people to be able to run our code.
[02:38.060 --> 02:42.160]  And further, we want people to be able to do that years, decades into the future.
[02:42.900 --> 02:44.040]  So how do we do that?
[02:47.960 --> 02:50.940]  Let's see, am I going back up?
[02:52.560 --> 02:58.120]  And okay, yeah, the other kind of problem here is that datasets are disappearing.
[02:58.780 --> 03:02.900]  You know, we're seeing federal cutting of spending.
[03:03.280 --> 03:04.760]  This example is from NOAA.
[03:04.920 --> 03:11.600]  So they had, I think, is it 17 or 18 datasets that if you go to the page for any of these, you just get a 404.
[03:12.060 --> 03:15.200]  These are no longer hosted by the government.
[03:15.200 --> 03:17.860]  And so a number of institutions have gone and backed them up.
[03:19.940 --> 03:22.140]  But, you know, it's obviously a problem.
[03:22.780 --> 03:29.840]  I like to see this as a bit of an opportunity where these are great datasets, but they aren't often linked to accessible examples.
[03:30.180 --> 03:35.900]  I don't have, you know, a good way to look at the shoreline coastline resource unless I know GIS really well.
[03:36.640 --> 03:40.220]  I is like, you know, an individual can actually interface with these datasets.
[03:40.580 --> 03:46.020]  So, you know, hopefully there's an opportunity here to actually build off of these.
[03:46.020 --> 03:48.920]  And, yeah, that's something we're going to talk about today.
[03:50.980 --> 03:55.360]  So, yeah, so goals in this framework, I guess, are...
[03:57.120 --> 03:58.240]  Let me pull this up.
[03:59.360 --> 04:03.640]  So, yeah, so I want to validate and communicate scientific knowledge first and foremost.
[04:04.400 --> 04:05.580]  Things should be reproducible.
[04:06.360 --> 04:09.020]  Things should be clear to people and clear to people at different levels.
[04:09.240 --> 04:12.560]  So, obviously, if you're reading the paper, you're in, like, academia, it should be clear.
[04:13.160 --> 04:21.240]  But also, if I, you know, stumble upon this paper and maybe interested in the field, it shouldn't be, like, a high barrier to get into that and run the code.
[04:22.040 --> 04:27.860]  You know, the first part of this is the code and data have to be available, which we're doing, but oftentimes it's not usable.
[04:28.600 --> 04:31.660]  And so there's lots of levels of usability and we'll kind of talk about that.
[04:32.300 --> 04:35.040]  And finally, we want to maintain some kind of historical records.
[04:35.340 --> 04:46.720]  So, you know, in the case of datasets no longer being present, you know, we're relying on this institution to host the datasets and to make them available.
[04:47.320 --> 05:02.680]  But, you know, we're also thinking about ways that we can allow individuals to do this or, you know, have a more decentralized approach where it's easy to upload datasets and archive and push research outputs to multiple archives.
[05:03.680 --> 05:07.840]  So, there are a few pretty good existing solutions for this.
[05:08.880 --> 05:11.080]  The first thing to highlight is CodeOcean.
[05:11.660 --> 05:21.080]  I think they've done a really good job in, like, a more SaaS commercial space of giving researchers tools for reproducibility.
[05:21.240 --> 05:26.240]  So you can have your whole experimental workflow in this kind of cloud environment.
[05:26.720 --> 05:32.460]  You can, like, click a button, have the whole thing rerun, and they take all your data and your code and output your figures.
[05:33.440 --> 05:39.240]  They do let you output some of the tools you need to actually build things locally.
[05:40.060 --> 05:42.140]  But you still have to build it yourself locally.
[05:42.420 --> 05:44.680]  And it's not open-source.
[05:45.320 --> 05:49.000]  So that's, you know, they're not maintaining their stuff anymore.
[05:49.240 --> 05:51.260]  They realize, okay, that we're not making enough money doing this.
[05:51.420 --> 05:53.680]  You know, it's kind of the end for that.
[05:54.420 --> 05:55.540]  BrainLife is really cool.
[05:55.540 --> 06:00.660]  They let you actually run, you know, their code on your own servers locally.
[06:00.660 --> 06:02.340]  So you can provide your own compute if you have it.
[06:03.320 --> 06:04.840]  But it is neuroscience specific.
[06:05.220 --> 06:10.900]  It's not really a general reproducibility computational framework at this point.
[06:11.900 --> 06:15.040]  And then this one called the whole tail is really interesting as well.
[06:15.800 --> 06:22.820]  I know it had, like, NSF funding, but it doesn't look like there's been a lot of movement on it in the last few years.
[06:22.820 --> 06:24.920]  All of their examples, like the links don't really work.
[06:25.160 --> 06:31.120]  And so it's great in theory, but I haven't actually seen like any outputs from it.
[06:32.700 --> 06:34.240]  So that's where we're at with that.
[06:34.680 --> 06:36.620]  And really what we're looking for is
[06:39.900 --> 06:42.860]  a framework that's open, modular, and easy to maintain.
[06:43.040 --> 06:44.480]  So it shouldn't be super complicated.
[06:44.660 --> 06:53.300]  It shouldn't be like, you know, required to be maintained by like a huge company to support the many existing and new research workflows that might pop up.
[06:54.260 --> 07:00.700]  So with all that background in mind, that is kind of why we came up with the ARTS open framework.
[07:00.940 --> 07:04.460]  So ARTS stands for archival, reproducible, and transparent science.
[07:06.340 --> 07:18.120]  We built this in the context of science, but I think it really applies to any any situation where you have some data, you want to do some computation on that data, and you want to produce something.
[07:18.700 --> 07:23.580]  And you want to be able to kind of archive that result and have it persist.
[07:24.220 --> 07:26.880]  So what I have up here...
[07:26.880 --> 07:28.560]  Oh, and I'm so sorry.
[07:30.000 --> 07:33.020]  I've gone to the next slide on my computer and not over here.
[07:33.380 --> 07:36.480]  So what I'm looking at is this slide.
[07:36.540 --> 07:46.360]  And you'll see here that there is a kind of standard research life cycle of collecting data, processing it into some derivative data.
[07:46.700 --> 07:49.320]  Oftentimes, this derivative data is something you can share more easily.
[07:49.320 --> 07:56.340]  The raw data might be protected health information, it might be terabytes, it might be really difficult to move around, etc.
[07:57.400 --> 08:03.020]  And then you have some analysis code that from that derivative data gives you some kind of result.
[08:03.260 --> 08:11.560]  Maybe that's a number, a table, it's some kind of figure, but you've created some scientific output from this data using code.
[08:12.200 --> 08:18.260]  And then you might write a paper or put some kind of manuscript together where you showcase the results and talk about them.
[08:18.820 --> 08:20.320]  So all these things get archived.
[08:21.380 --> 08:27.880]  And a big part of what we're proposing is to essentially have better practices around how we do this.
[08:28.160 --> 08:36.680]  So running code in containers, having a kind of clear environment that things run in, also using timestamping, we'll get into all these details.
[08:37.540 --> 08:43.080]  But this is kind of how people are doing things now in a lot of ways across many fields.
[08:43.360 --> 08:51.020]  And so we're just looking at ways to kind of create some best practices that people can kind of speak to each other in within that.
[08:52.300 --> 09:06.200]  And so here are kind of seven topics that if you're able to touch on all of these inputting your research output out, you'll have something that's pretty much maximally compatible with this open framework.
[09:09.340 --> 09:13.060]  Really, the only ones that are mandatory are the first one, the last one.
[09:13.220 --> 09:18.000]  So it's really important that you have a license with whatever you put out in most jurisdictions.
[09:18.000 --> 09:26.100]  It's also really important that you have some kind of configuration file, and we'll get into those, just so that there is a standard of some sort.
[09:26.360 --> 09:28.480]  All the other pieces are really great to have.
[09:28.720 --> 09:34.820]  The more you do, the more accessible, the more persistent, the more reproducible your output will be.
[09:35.400 --> 09:38.160]  But none of them are strictly mandatory.
[09:39.520 --> 09:41.280]  So we'll start with licensing.
[09:42.640 --> 09:45.500]  Oh, and actually, yeah, first, we do have a paper out on this.
[09:45.700 --> 09:48.280]  The first version is up on archive, if you are curious.
[09:48.920 --> 09:51.300]  I submitted a second version recently.
[09:51.300 --> 09:53.120]  It should be out on Tuesday, probably.
[09:54.220 --> 09:57.340]  But this is like, you know, if you want to read even more.
[09:58.340 --> 10:00.740]  So yeah, the first thing I want to talk about is licensing.
[10:02.380 --> 10:05.040]  Here's a comparison of some open-source licenses.
[10:08.100 --> 10:13.220]  So in general, it's, you know, copyright is default in most jurisdictions.
[10:13.220 --> 10:17.660]  What that means is that if you just put code out, it is kind of in a proprietary situation.
[10:17.660 --> 10:24.660]  No one can actually use that code unless you provide like a license and define terms for how it can be used.
[10:25.140 --> 10:27.540]  The same actually applies for data too.
[10:27.840 --> 10:31.840]  So if you just put data out, it is important to have a license with that if you can.
[10:32.500 --> 10:42.240]  Creative Commons has some really great, hopefully everyone's familiar with that, some great licenses for attribution and share like terms.
[10:44.940 --> 10:48.540]  So in the ARTS open framework, it's great if you can have an open-source license.
[10:48.680 --> 10:52.900]  And these are some examples of different open-source licenses, but you also don't need to.
[10:52.980 --> 10:54.100]  You can use non-commercial licenses.
[10:54.160 --> 10:56.720]  You can use a proprietary license and that's compatible with ARTS.
[10:58.480 --> 11:01.020]  You know, I just personally would love if you could open-source it.
[11:01.020 --> 11:03.600]  So that's licensing.
[11:05.400 --> 11:11.720]  In terms of archives, see if we can get to the next slide.
[11:16.200 --> 11:17.280]  Not quite.
[11:18.060 --> 11:19.020]  There we go.
[11:19.580 --> 11:26.540]  So in terms of archives, there are a ton of different existing archives that researchers often publish their data to.
[11:28.780 --> 11:33.680]  Usually the guidance is that if you can find an archive that fits your specific use case.
[11:33.860 --> 11:44.020]  So if you are in, say, you know, you work for NOAA and you're putting out a data set to do with ocean temperatures, there might be a specific data repository for data sets like that.
[11:44.180 --> 11:49.320]  You should put it in there because that's the most kind of findable place for that data.
[11:49.860 --> 11:52.940]  These are more general use archives.
[11:53.240 --> 11:54.060]  So they're good as backups.
[11:54.240 --> 11:58.240]  They're also good if you can't have a specific data storage for your data set.
[11:59.040 --> 12:06.740]  What you'll notice here, so we're kind of judging these archives on how accessible, persistent, and trusted they are.
[12:06.900 --> 12:16.620]  Those are kind of subjective terms, but there are a bunch of different points under that which might make an archive more or less, say, accessible.
[12:17.720 --> 12:21.060]  None of these archives actually hit all the points.
[12:21.420 --> 12:23.100]  So the closest one would probably be Zenodo.
[12:23.640 --> 12:26.500]  A lot of data sets are stored on Zenodo right now.
[12:26.660 --> 12:31.480]  It integrates really nicely with ORCIDs and other kind of identifiers.
[12:32.040 --> 12:33.980]  You can mint a DOI on Zenodo.
[12:34.760 --> 12:41.920]  The main thing that Zenodo is missing is some way to actually programmatically pull data.
[12:41.920 --> 12:47.680]  So if I have a repository there, I can't use version control to clone it down.
[12:47.800 --> 12:50.600]  I can't look at individual pieces of that data set.
[12:50.700 --> 12:57.200]  I need to kind of take the whole tar file or zip file and download it, open it up.
[12:57.320 --> 13:04.060]  And so it's not ideal for doing things in an automated way or doing things in a reproducible way.
[13:04.940 --> 13:07.900]  But interestingly enough, the Internet Archive is really good for that.
[13:08.040 --> 13:14.000]  But it isn't really built for having provenance and history over time.
[13:15.340 --> 13:19.140]  Everything's kind of mutable and the original uploader can delete their data set.
[13:19.280 --> 13:23.620]  They can modify it without a change log.
[13:24.360 --> 13:27.540]  So that's a problem for things we want to persist.
[13:27.540 --> 13:36.400]  But the back end of it actually does allow you to clone things down and have containers and stuff.
[13:37.880 --> 13:39.240]  So that's archives.
[13:41.300 --> 13:44.740]  You know, we talked a little bit about, or just touched on now, version control.
[13:44.980 --> 13:45.840]  I use Git primarily.
[13:47.860 --> 13:52.320]  But the Arts Open Framework only dictates that you use some kind of version control.
[13:54.180 --> 14:04.920]  And, you know, you can have multiple repositories, you can have multiple different pieces of this, but it's important in order to preserve some kind of stock record that you are using version control.
[14:08.140 --> 14:18.060]  The other consideration here is that for any kind of science or preserving records, it's good to use a merge strategy over a rebase strategy if you're doing version control.
[14:18.560 --> 14:21.460]  That way, you know, rebase generally rewrites history.
[14:22.260 --> 14:23.980]  So just something to watch out for.
[14:26.760 --> 14:34.360]  And then, yeah, so a big piece of what we're actually trying to do here is, I guess, get people to use containers where they can.
[14:35.020 --> 14:38.540]  If we can do like a brief show of hands, how many people are familiar with containers?
[14:38.860 --> 14:39.700]  Do you use containers?
[14:40.640 --> 14:42.660]  Looks like most folks, some folks.
[14:43.760 --> 14:51.840]  So containers are, yeah, essentially standalone, executable packages, and they include everything that an application will need to run on your system.
[14:52.660 --> 14:56.160]  They are maybe similar to virtual machines.
[14:56.400 --> 15:09.820]  If you haven't heard of containers, virtual machines are sort of like an older but similar technology that let you virtualize a whole computer in a different operating system, different, yeah, different operating system.
[15:09.960 --> 15:14.220]  And then you can run it on maybe the computer in front of you or when you have access to remotely.
[15:16.020 --> 15:20.700]  So containers essentially are a big piece of allowing things to be reproducible and sharing environments.
[15:20.800 --> 15:25.620]  If I run something in one environment on my machine, I can send you a container file.
[15:25.620 --> 15:35.420]  And if you're running the same container engine, you can run that binary and essentially not have any environment setup time.
[15:35.660 --> 15:36.740]  And it makes things a lot easier.
[15:37.240 --> 15:41.580]  So one distinction I want to make is between a container file and a container image.
[15:42.040 --> 15:45.080]  So I like to think of this as like the recipe and the cake.
[15:45.260 --> 15:48.840]  If you have a container file, that's the recipe, but you still have to make the recipe to build it.
[15:49.280 --> 15:51.180]  And the container image is the actual cake.
[15:51.560 --> 15:53.280]  The recipe is really easy to pass around.
[15:53.280 --> 15:56.580]  It's just, you know, maybe a few lines of text.
[15:56.820 --> 16:00.320]  Whereas the image could be, it's a whole compiled binary, it might be three gigs.
[16:01.060 --> 16:04.480]  So passing around container files is nice.
[16:04.760 --> 16:19.620]  But if you're able to actually compile a container image, people can download that and run everything that you've been able to put together without building on their own platform, which is a place where a lot of errors and issues are introduced.
[16:20.220 --> 16:21.800]  So I mentioned CodeOcean before.
[16:22.100 --> 16:25.900]  They let you download a container file, but not the container image.
[16:26.020 --> 16:28.940]  So you still would have to go and build whatever outputs yourself.
[16:30.480 --> 16:36.560]  Some popular container engines are Podman, Docker, and AppTainer, which used to be called Singularity.
[16:37.160 --> 16:47.900]  AppTainer is really popular in scientific communities because you can run it on open on-demand clusters and compute infrastructure that universities already have.
[16:48.720 --> 16:55.320]  But, you know, Podman and Docker are maybe a little bit more ubiquitous within individual spheres.
[16:55.960 --> 17:01.140]  The nice thing about these engines is that they conform to a shared spec.
[17:01.540 --> 17:08.380]  So if, you know, Docker goes out of fashion in five years, the spec is still there, you can build something on top of it.
[17:09.100 --> 17:10.860]  They're all kind of speaking the same language.
[17:11.560 --> 17:16.020]  One big issue with using containers, especially in a research environment, is they can be really large.
[17:16.200 --> 17:25.620]  Like I mentioned, that container image could be up to like 10 gigs if you're pulling a lot of different packages and want to be doing, you know, some kind of complex analysis.
[17:27.120 --> 17:28.380]  So that's just a consideration.
[17:28.720 --> 17:30.660]  There are some strategies to deal with that.
[17:31.940 --> 17:33.180]  But it's good to know.
[17:36.720 --> 17:39.560]  One, yeah, so to keep going with pieces of the framework.
[17:40.560 --> 17:41.960]  The next piece is Providence.
[17:42.180 --> 17:44.320]  So it's how do we actually keep that historical record.
[17:45.060 --> 17:47.880]  And there's a technology called Trusted Timestamping.
[17:48.040 --> 17:50.860]  It's defined in RFC 3161.
[17:52.220 --> 18:02.520]  My longtime mentor who I work with put out recently a Trusted Timestamping website that allows you to upload a piece of data in your browser.
[18:02.680 --> 18:04.300]  It doesn't send any data anywhere.
[18:05.160 --> 18:11.320]  And it'll essentially hash that data, send the hash of your data to a bunch of different timestamp authorities.
[18:12.060 --> 18:17.280]  And then this, you know, all these hashes kind of get put together and hashed and put in a file.
[18:18.100 --> 18:26.100]  So what this allows you to do is over time essentially prove that a certain piece of data was collected when you say it was.
[18:27.020 --> 18:31.940]  And then build a record that, you know, spans, yeah, potentially years or more.
[18:32.640 --> 18:36.980]  And one thing to note with this is if certificates expire, right?
[18:37.040 --> 18:40.840]  So if you have a timestamping authority, it'll have some certificate that'll have an expiry date.
[18:41.360 --> 19:00.260]  As long as you are essentially hitting that timestamp authority at some regular interval, you can hash the entire record and put that in sort of the future and make a sort of trusted provenance that way.
[19:00.740 --> 19:09.400]  So all to say there are ways to ensure that you have continuous kind of trusted data over time.
[19:09.640 --> 19:23.540]  The really nice thing about this is when mixed with something like Jupyter Notebook, you can actually have like a verifiable electronic lab notebook that you version control over time and you can prove that things happened and data was collected when you say they were.
[19:24.420 --> 19:26.380]  So yeah, this is a great technology.
[19:26.840 --> 19:29.400]  I would encourage people to use it and it's really easy to plug into Git.
[19:29.780 --> 19:34.200]  You can just use a post commit kind of script and it just works.
[19:34.620 --> 19:35.060]  So that's nice.
[19:37.500 --> 19:37.900]  Cool.
[19:39.180 --> 19:40.880]  Getting towards the end here of the pieces.
[19:41.340 --> 19:41.740]  Documentation.
[19:41.740 --> 19:55.560]  I think this is the most important part because even if you have the license and I can, you know, access your code and your data, I don't know what's happening if you don't have documentation, especially documentation multiple levels.
[19:55.780 --> 20:03.180]  So you, you know, ideally have high level documentation that is accessible for lots of, you know, kind of a general audience, you have documentation, your code.
[20:03.360 --> 20:09.780]  I think this is a lot to ask in the research community because oftentimes it's like one grad student and they're pretty stretched.
[20:10.060 --> 20:12.560]  And so to add more work onto that is difficult.
[20:12.900 --> 20:23.800]  But at the very least, when you're putting something out and you want it to be reproducible, it's really important to have some kind of instructions on how to set it up, how to run it, and then how to use or change the outputs.
[20:25.100 --> 20:26.660]  So that's a big piece.
[20:27.540 --> 20:33.760]  And then finally, the end of the spec is this basically how you set things up.
[20:33.860 --> 20:36.920]  So the file structure and then including this config.n file.
[20:37.500 --> 20:43.080]  So as I've been saying, the ARTS open framework is kind of unopinionated on purpose.
[20:43.480 --> 20:51.200]  We don't want to have a ton of mandated ways of doing things.
[20:51.220 --> 21:08.940]  We want to remain flexible to support the different kinds of research workflows that are present and essentially present a language that we can all speak and standardize the ways that code and data play with each other.
[21:09.800 --> 21:13.960]  So this is an example of a very minimal file structure that has code.
[21:14.060 --> 21:14.680]  It has data.
[21:15.060 --> 21:20.900]  It also includes under this end directory, a container file and a compiled image that you could run.
[21:21.120 --> 21:28.600]  It of course, includes a license and then has some scripts that you could run locally if you wanted to without the container.
[21:29.640 --> 21:33.600]  None of this stuff is really necessary except like I said, the license and the config.n file.
[21:33.800 --> 21:40.240]  And in the config.n file is where you specify where the different pieces of your deposition can be found.
[21:40.260 --> 21:42.220]  So the code, the data, etc.
[21:42.780 --> 21:44.240]  So we'll see an example of this.
[21:44.900 --> 21:50.280]  But this is sort of the spec we're defining and it's open.
[21:51.480 --> 21:53.400]  We don't really have the paper.
[21:53.480 --> 21:55.420]  We don't really have like a spec definition.
[21:55.700 --> 21:59.640]  So at some point, we'll put that out and it'll be open-source and people can make pull requests.
[21:59.860 --> 22:02.740]  So it can also be sort of a living scaffold.
[22:04.560 --> 22:05.000]  Great.
[22:05.240 --> 22:10.040]  So the kind of toy example I set up to test this is this watch calibration example.
[22:10.640 --> 22:14.400]  So I bought a cheap watch movement.
[22:14.480 --> 22:17.480]  I think it was like 20 bucks online.
[22:17.480 --> 22:18.820]  It's a mechanical watch movement.
[22:18.960 --> 22:23.500]  If you hold up to your ear, you can hear like a little tick frequency at approximately six hertz.
[22:24.120 --> 22:27.480]  So we can do is you can rest on your laptop, you can record it.
[22:28.680 --> 22:33.560]  And then I wrote some code in Python to analyze it, essentially see the drift of that clock over time.
[22:33.560 --> 22:38.040]  So I can say, okay, over a month, how far out of sync is my clock going to be?
[22:38.040 --> 22:40.460]  And I can adjust that.
[22:40.760 --> 22:53.460]  So, you know, this example doesn't have anything to do with the research that we do, but it seemed like a good test of recording audio data, which is fairly large, and then doing analysis with signal processing and Python.
[22:54.780 --> 23:01.740]  And then, yeah, so the test of this is, can we archive this and sort of make it accessible and runnable on a lot of different platforms?
[23:03.860 --> 23:05.060]  Let's see what we're doing on time.
[23:05.260 --> 23:05.980]  Pretty good on time.
[23:06.220 --> 23:07.480]  So we'll take it slow.
[23:09.860 --> 23:10.260]  Cool.
[23:10.500 --> 23:11.700]  No, not time for questions yet.
[23:11.920 --> 23:12.020]  Okay.
[23:13.500 --> 23:19.000]  Yeah, so the main output of this deposition is essentially the environment.
[23:19.180 --> 23:25.600]  It's the fact that I put a Jupyter notebook together, I have all my code and an explanation of what's happening.
[23:26.400 --> 23:30.760]  And this is kind of where I did a lot of the analysis for this experiment.
[23:31.160 --> 23:42.380]  And I want other people to be able to just dive in and start seeing that they should be able to run the code, they should be able to see the actual graphs being made, they should be able to modify the code slightly and see how that changes things.
[23:43.920 --> 23:46.060]  I'm a very hands-on learner, that works for me.
[23:46.780 --> 23:48.220]  That might not work for everybody.
[23:48.400 --> 24:00.860]  But I think when you are producing something that's often static and just in a PDF file, it's a lot more powerful to have it be configurable and kind of manipulable like this.
[24:01.840 --> 24:07.640]  So this is running in a container in my machine, I took a screenshot, and this is kind of the goal is to make this accessible.
[24:09.360 --> 24:09.800]  Great.
[24:10.620 --> 24:23.000]  So this deposition, to kind of go back to the framework, we use GPL and Creative Commons share like attribution licenses.
[24:23.780 --> 24:26.640]  So, and again, I'm not on the right slide.
[24:26.840 --> 24:27.180]  Here we go.
[24:28.640 --> 24:31.120]  So yeah, so those are the licenses we picked.
[24:31.320 --> 24:36.040]  That means you can use distribute these if you make changes.
[24:37.360 --> 24:39.780]  We ask that you keep those licenses in place.
[24:41.540 --> 24:43.900]  The version control system we're using for all this is Git.
[24:44.120 --> 24:48.060]  So you can get clone the deposition from GitHub.
[24:48.660 --> 24:54.020]  It's also archived on a couple of different archive platforms, but it is nice to be able to clone it.
[24:54.840 --> 25:01.420]  So yeah, the archives are Stanford's digital repository, which is really great.
[25:01.580 --> 25:13.240]  And if we go back to that archive table, there are a lot of points that it is really kind of doing well on, but it doesn't allow just anyone to upload.
[25:13.340 --> 25:16.540]  You have to be collaborating with people at Stanford.
[25:16.540 --> 25:18.280]  So it's obviously not very accessible that way.
[25:18.500 --> 25:30.580]  And then of course, it's on Zenodo, which is probably the most successful platform, but doesn't have some of the fun backend ways of pulling the files automatically, like I said.
[25:31.900 --> 25:38.160]  So I'm going to walk through a couple of the files in the deposition to give you a sense of how this looks in practice.
[25:39.700 --> 25:46.020]  The first, and we've been talking about containers a bunch, but it's probably not super concrete, is the actual container file.
[25:46.640 --> 25:53.600]  And so this is the environment, essentially a definition of the environment that I'm using to run my analysis in.
[25:53.600 --> 25:58.060]  You can see I'm using Python version 3.10.
[25:58.560 --> 26:04.540]  This is sort of a Linux environment, like a Debian flavor.
[26:04.860 --> 26:14.400]  And so I'm using apt-get to essentially install the audio and other dependencies I need.
[26:14.620 --> 26:29.100]  And then I'm basically just creating a directory to run my code in and installing all the Python requirements, copying the files over, and installing the actual package.
[26:29.500 --> 26:32.880]  So I've written the code itself as a Python package.
[26:34.480 --> 26:44.900]  There are a bunch of entry points with Bash, but it's a fairly simple definition that then, when compiled out, allows other people to jump in and start coding right away.
[26:53.510 --> 26:55.350]  The data providence piece.
[26:55.690 --> 26:59.170]  So this is what that trusted timestamping file looks like.
[27:01.490 --> 27:06.830]  Essentially, there's a hash of the actual data, it tells you what algorithm it's using, it gives you a name for the data set.
[27:07.650 --> 27:12.030]  And you'll see at the bottom here, there's a list of timestamp authorities it hits.
[27:12.130 --> 27:13.850]  I think it hits six or seven at the moment.
[27:14.470 --> 27:19.890]  So it gets one from DigiCert, it gets a reply from that, and it provides a hash.
[27:20.090 --> 27:34.350]  And so it's important to provide these timestamp JSON files with your deposition when you actually archive it somewhere, because that's how you prove that essentially the data I collected and the code I wrote happened when I said it happened.
[27:36.610 --> 27:48.130]  Someone can basically then go and use these timestamping tools to verify that by uploading the data and uploading this JSON file and essentially checking that, okay, these match and all the hashes work together.
[27:55.050 --> 27:57.790]  The other piece I said is really important is having documentation.
[27:58.170 --> 27:58.950]  So we have a readme.
[28:00.210 --> 28:06.730]  You'll see here essentially this is talking about how to run the scripts that essentially give you these container entry points.
[28:06.930 --> 28:12.170]  So you can generate the figures again, if you run the run script with this generate figures command.
[28:12.470 --> 28:23.150]  We go in and load all the dependencies, run the whole experiment, and then output all the figures that some of which you saw in this presentation.
[28:23.430 --> 28:28.610]  So they're all kind of baked into this framework and able to be run again and again.
[28:29.770 --> 28:30.110]  And then
[28:33.570 --> 28:35.890]  finally, there's this configuration file.
[28:36.290 --> 28:46.590]  So here we basically have a number of art specific environment variables.
[28:46.870 --> 28:49.610]  And this is the part that, I guess, is the actual standard.
[28:49.810 --> 28:55.950]  So this is something that, you know, if you want to be kind of in line with this framework, you will do need to define these variables.
[28:56.470 --> 28:57.590]  But they're fairly simple.
[28:57.790 --> 29:10.510]  They define where the raw data is, if it's present, where the derivative data is, if it's present, any code, and you can list multiple directories for this potentially, any environment path.
[29:10.710 --> 29:18.290]  So in this case, we just have the container file and the container image just in the root directory of the deposition.
[29:18.290 --> 29:20.430]  So we can just list the present directory here.
[29:21.890 --> 29:23.030]  Any outputs you have.
[29:23.230 --> 29:27.430]  So, you know, I've put some outputs in this figures directory because I had a bunch of figures with this.
[29:28.490 --> 29:35.110]  And then, you know, specifically the license, the readme, and scripts you might want to supply, those should also be present.
[29:37.390 --> 29:43.450]  The other cool thing about this is you can actually provide high-level knobs to your experiment in this file itself.
[29:43.730 --> 29:52.470]  So in this case, when I'm actually running data collection scripts, if I want to run data collection for five seconds, that's specified here in the recording duration.
[29:52.630 --> 30:05.810]  I can just change this to 60 seconds and essentially run the scripts again, and that'll give me a one-minute long recording with the exact same environment, the exact same parameters that the original experiment was running, just with a different recording duration.
[30:06.550 --> 30:14.010]  And by the same token, you can provide variables for the analysis that are kind of high-level knobs.
[30:14.650 --> 30:28.010]  Of course, you can always go into the code and change these, but I think providing these at a high level is really useful for people who want to enter into a new field or maybe are in the same field, but just don't work in the exact same environment that you do.
[30:30.030 --> 30:35.090]  So the last piece of this is essentially a demo.
[30:37.610 --> 30:40.470]  So I guess we can open this up because we have a little bit of time.
[30:41.350 --> 30:44.910]  But there's this really great tool called MyBinder, Binder Hub.
[30:45.410 --> 30:51.910]  What they do is they essentially take a Git repository.
[30:51.910 --> 30:57.410]  So you can put in a link to a branch in a specific file, some kind of notebook.
[30:57.690 --> 31:02.070]  And what it will do is it'll go and it'll pull the container image from that repository.
[31:02.790 --> 31:06.410]  It'll... or I actually believe it pulls the container file.
[31:06.590 --> 31:08.150]  So that does actually do a full build.
[31:08.270 --> 31:18.010]  So it pulls the container file, it builds the image, and then it hosts it with some amount of memory and CPU resources for you.
[31:18.650 --> 31:29.030]  The really great thing about this is that I can go and share that link and someone else can essentially work in my environment that I just set up without any setup on their own machine.
[31:30.870 --> 31:34.190]  So here is the demo.
[31:35.390 --> 31:36.950]  I'll put this over here.
[31:37.470 --> 31:39.090]  We'll see if this works right now.
[31:39.330 --> 31:47.530]  But essentially, we're launching this container that I had.
[31:49.770 --> 31:50.390]  Cool.
[31:50.690 --> 31:55.670]  So while that does its thing, I guess we're into it.
[32:00.270 --> 32:03.430]  And this is basically pulling up a JupyterLab server.
[32:03.690 --> 32:06.350]  So I'm using Python.
[32:06.930 --> 32:09.310]  You can use other programming with this as well.
[32:09.690 --> 32:14.730]  But you'll see here that this is the notebook that I had up earlier, the one that I ran locally.
[32:14.730 --> 32:18.170]  It's running just on a remote machine right now.
[32:19.010 --> 32:26.310]  And I can actually go, and if I can find my mouse somewhere on the screen, I can go and run this code.
[32:26.910 --> 32:28.530]  You can see it thinking over here.
[32:28.730 --> 32:40.550]  And boom, it ran, it imported all of the Python dependencies I had, including matplotlib and scipy, which are kind of these powerhouse tools for doing analysis.
[32:41.090 --> 32:52.550]  And then I can go and load my data and actually kind of regenerate the figures that I actually had in the presentation and put in my paper.
[32:52.550 --> 32:58.530]  So yeah, this is kind of how I want to keep doing research in the future.
[32:59.030 --> 33:08.270]  And like I said, I think this is applicable not just for research, but if you're doing any kind of project and have data and code, it's just a different way of being able to share that.
[33:09.550 --> 33:14.030]  So I think we're a little early on time, but I am out of slides.
[33:14.490 --> 33:18.330]  So yeah, if people have questions, open to taking those.
[33:18.970 --> 33:24.090]  And yeah, if you want any of the links, I'll put the link to the presentation up.
[33:24.730 --> 33:27.210]  That's on the first slide and people can look at that.
[33:27.990 --> 33:28.810]  So yeah, thanks so much.
[33:39.190 --> 33:40.870]  Pull up the slides one more time.
[33:44.450 --> 33:44.950]  Hey.
[33:47.310 --> 33:47.810]  Yeah.
[33:48.750 --> 33:49.810]  I'm a librarian.
[33:49.890 --> 33:55.250]  So I have colleagues who help people with data management plans and that kind of thing.
[33:55.470 --> 33:55.750]  Cool.
[33:55.930 --> 33:56.530]  I'm curious.
[33:56.570 --> 34:12.630]  I know that in terms of open data, there have been some real big sort of carrot and stick changes in the last few years in terms of grant requirements for sharing data and then publications.
[34:12.630 --> 34:16.310]  So journals requiring that that data be open.
[34:16.550 --> 34:24.530]  I'm curious if you have thoughts about how this framework could get integrated into that or because I know there's a lot of issues.
[34:24.770 --> 34:27.490]  People assume, okay, I've got my data is up somewhere.
[34:27.810 --> 34:28.590]  I'm done.
[34:28.790 --> 34:30.870]  I don't have to help anybody use it.
[34:32.030 --> 34:32.390]  Definitely.
[34:32.390 --> 34:32.750]  Yeah.
[34:32.830 --> 34:33.670]  Thanks so much for that question.
[34:34.230 --> 34:39.390]  The federal mandates on that have actually been a big reason why we kind of decided to do this right now.
[34:40.950 --> 34:45.890]  The kind of the status of that, I think we're a bit unsure of right now, obviously.
[34:47.050 --> 34:53.670]  But we were under the assumption that, okay, by this year, we essentially need to, at least for federally funded things, have the data available.
[34:53.930 --> 34:57.270]  And if we're going to do that, we should do it the right way and have also usable.
[34:57.890 --> 35:08.790]  I hear you where, you know, professors, students are overworked and they don't have the time to do extra work to make the data more usable, more shareable.
[35:09.210 --> 35:12.030]  And so that is a big problem.
[35:12.230 --> 35:17.690]  I think I've definitely seen funding towards like research software engineers.
[35:17.910 --> 35:19.930]  And like, I think there's a lot of cool work happening there.
[35:20.090 --> 35:22.530]  And that's like one place that that can happen.
[35:22.530 --> 35:26.170]  Um, but yeah, I think you raise a really good point.
[35:26.410 --> 35:45.170]  And I guess my hope was something like this is that we start putting out outputs and, you know, labs and communities where we do have the resources to work like this, um, where it's just so easy to play with the data and build on things.
[35:45.250 --> 35:47.030]  And it just, that's a good example.
[35:47.250 --> 35:49.690]  And so I guess that is more of the carrot approach.
[35:49.950 --> 35:53.490]  Um, because the six networks are kind of all over the place right now.
[35:53.930 --> 36:00.410]  Um, so I don't know if I answered the question, but I'm, yeah, I'm hoping to just provide really nice carrots that are easy to use.
[36:00.730 --> 36:04.950]  And, and like I said, I think some of those tools like code ocean, for example, is a really nice one.
[36:05.330 --> 36:11.690]  Um, but it's also like a big piece of infrastructure and it's like adopting that as a whole separate thing.
[36:11.970 --> 36:14.690]  Um, so that's Yeah.
[36:15.150 --> 36:25.450]  Um, so I think one of the, what you were, your answer just made me think about how some of the sharing data stuff, there have always, there have been some fields where people always have done that.
[36:25.610 --> 36:33.390]  Um, and computer science comes to mind as one place where a lot of people are sharing things on their own website in computer science.
[36:33.790 --> 36:39.030]  Um, but a lot of scholars see their data as a proprietary thing.
[36:39.030 --> 36:45.370]  So even if they're required to share their data, they may not want to share how to use the data the way they did.
[36:45.550 --> 36:45.810]  Yes.
[36:46.090 --> 36:53.150]  Um, so I'm curious what discipline you've come from or you're, you're sort of where you're finding this is more popular.
[36:53.290 --> 36:55.890]  You gave some science, some sort of medical examples.
[36:56.650 --> 36:57.250]  Um, definitely.
[36:57.590 --> 37:00.770]  Um, it's, it's very applicable anywhere there is federal funding, right?
[37:00.910 --> 37:04.910]  Just because you are required to share data and put it in the public domain oftentimes.
[37:05.350 --> 37:11.090]  Um, which is also interesting because sometimes that's not the right license for things, even though, um, anyway.
[37:11.530 --> 37:19.930]  Um, yeah, I think the fields that I've been reading up on that already have a lot of really cool examples and work have been done, work has been done here.
[37:20.210 --> 37:23.030]  Definitely astrophysics, um, a lot of life sciences.
[37:23.370 --> 37:25.810]  So genomics, um, I come from systems neuroscience.
[37:25.890 --> 37:29.350]  So we have a lot of data, um, as well.
[37:29.550 --> 37:40.830]  And then I think on the climate side, because things are so federally funded, um, I'm sure there are more and I'm missing some, um, computer science makes a lot of sense because everyone is already using all these tools.
[37:41.010 --> 37:42.790]  They're already using version control and containers.
[37:42.790 --> 37:45.730]  And it's just like, why would I not put my research out that way?
[37:46.510 --> 37:50.570]  Um, so yeah.
[37:50.770 --> 37:51.970]  Sorry, what was the question?
[37:52.290 --> 37:53.750]  No, you've answered it.
[37:53.850 --> 37:56.110]  Yeah, I was just curious what disciplines you've worked with.
[37:56.110 --> 37:56.550]  Yeah.
[37:56.610 --> 37:57.010]  Yeah.
[37:57.130 --> 37:58.830]  Those are, those are the disciplines, I guess.
[37:59.150 --> 37:59.310]  Yeah.
[37:59.370 --> 37:59.830]  Thanks so much.
[38:07.450 --> 38:18.270]  So I had a, I was thinking back to work, working in a lab and having to keep a lab notebook and hating the, like, you know, hating this like extra thing to do that broke flow.
[38:18.830 --> 38:36.370]  But then it not only became useful from like a record keeping standpoint, but from like, for external purposes, but also for me, like I went back and looked at my data and data, I probably, I might otherwise not have written down because it seemed, um,
[38:37.370 --> 38:39.390]  not core to what I was doing at the moment.
[38:39.550 --> 38:56.170]  So I guess I was curious if you've seen similar value generated by this tool where, um, it's not just for external compliance reasons, but, um, promotes internal record keeping in a way that can, um, help science move along anyways.
[38:56.830 --> 39:01.410]  And I guess, I just didn't know if there were examples of that or if you've seen that, uh, in using this.
[39:02.230 --> 39:08.470]  Um, sorry, the question is, um, like examples of, uh, yeah, I guess like, have you seen this be useful?
[39:08.630 --> 39:14.650]  Are there incentives for people to use this on their own even without external compliance, uh, purposes?
[39:15.030 --> 39:15.090]  Yeah.
[39:15.430 --> 39:16.190]  Yeah, definitely.
[39:16.730 --> 39:18.670]  Um, oh, thanks for the question.
[39:19.910 --> 39:28.070]  I, I mean, for my own records, like I was saying, I definitely will come across things where I wrote code a few years ago and I have no clue how to run it and everything's broken.
[39:28.470 --> 39:31.570]  And if I like just put it in container, like that problem wouldn't exist.
[39:32.210 --> 39:37.870]  So yeah, for myself, I have started to create that habit and it's been useful.
[39:38.710 --> 39:46.650]  Um, yeah, I think, um, it is that mix of like mandates and like what's practical.
[39:47.090 --> 39:52.370]  And so, um, one thing we really want to do, and I guess this kind of goes back to the, the first questions as well.
[39:52.650 --> 40:00.390]  Um, but what we really want to do in at least our specific field of neuroscience is provide a bunch of containers that we use day to day.
[40:00.550 --> 40:08.630]  We have containers that on our clusters, we just, we SSH into a machine that containers are already set up and it's just there and those should be available.
[40:08.850 --> 40:12.330]  People should be able to use the same compute environments that we use.
[40:12.690 --> 40:16.350]  Um, and so, you know, that's something we're working on and, um, hope to have out.
[40:16.530 --> 40:21.330]  Um, you know, we'll see you on, but, uh, hope to have out at some point.
[40:21.670 --> 40:28.070]  So, um, I think part of that is standardizing the kinds of compute environments we use.
[40:28.230 --> 40:35.930]  So if you are really opinionated on a specific field or, or kind of application, cool, like, what's your environment?
[40:36.090 --> 40:36.710]  Like, what are you using?
[40:37.050 --> 40:39.870]  What's, is there a container file I can download and run on my computer?
[40:40.050 --> 40:44.810]  Because then I can sort of use the same tools that you're using and I don't have to go through the whole setup.
[40:45.530 --> 40:51.630]  Um, so I think we still have a culture of, um, having a list of dependencies and installing those.
[40:51.870 --> 40:54.570]  And, um, as we saw yesterday in the workshop, right?
[40:54.730 --> 40:56.250]  It works on different machines, different ways.
[40:56.670 --> 41:10.830]  And, um, um, I think if we're able to shift to a culture of having more of these shared environments, um, it'll make a lot of those things easier and it'll make getting into new fields and looking at different data and more accessible.
[41:11.390 --> 41:13.710]  Um, so that's, I guess that's the hope.
[41:19.830 --> 41:20.130]  Cool.
[41:20.790 --> 41:25.630]  Um, it doesn't look like we have more questions, but yeah, now's your time if you have one.
[41:26.410 --> 41:27.150]  You're good at the time.
[41:27.150 --> 41:27.930]  Okay.
[41:28.490 --> 41:28.950]  Thanks.
[41:29.150 --> 41:29.670]  Well, thanks everyone.
[41:29.790 --> 41:30.250]  Thanks so much.
[41:30.850 --> 41:31.450]  Thank you.