Latest / No Way Out / The Wayback Mission: Mark Graham on Preserving the Past with the Internet Archive
Transcript
- 0:20Mark Graham: The man who came up with Frank Burns.
- 0:23Mark Graham: Colonel, frank Burns was a mentor of mine and Frank was part of that team.
- 0:28Mark Graham: He was the guy who came up with the phrase be all you can be, be all you can be.
- 0:33Brian "Ponch" Rivera: yeah, which the Army you know, that was the greatest.
- 0:36Brian "Ponch" Rivera: That was the most well-known ad slogan at the time that they discontinued it for an Army of One, which was unbelievable.
- 0:43Mark Graham: That's really something.
- 0:44Mark Graham: An army of one which was unbelievable.
- 0:45Brian "Ponch" Rivera: That's really something.
- 0:46Brian "Ponch" Rivera: Yeah, there was a General Stubblebine that was an intelligence officer, that was you can start recording this.
- 0:54Mark Graham: You are recording it actually, right?
- 0:55Mark Graham: Oh, yeah, we are Okay.
- 0:56Brian "Ponch" Rivera: Yeah, we are because it got too interesting, it did.
- 0:59Mark Graham: Very quickly.
- 1:00Mark Graham: Well, I don't know Stubblebine, but the person I'm most familiar with is Jim Shannon, and Jim Shannon was the creator of the First Earth Battalion and I have some of their paraphernalia.
- 1:28Mark Graham: Like I say a yes, okay, so people do that.
- 1:28Mark Graham: I'm pretty sure we have First Earth Battalion material there.
- 1:29Mark Graham: If we don't, I'll definitely have to remedy that.
- 1:30Mark Graham: Maybe someone could check that.
- 1:32Mark Graham: But there was the main text that described the theories.
- 1:38Brian "Ponch" Rivera: Yeah, that's the kind of stuff that Punch and I certainly talk about.
- 1:40Brian "Ponch" Rivera: I mean, we talk about a lot of things that would be not mainstream, I guess, when it comes to military strategic thought, maybe like the incorporation of, say, psychedelics to treat PTSD, or you know he didn't know himself was such a controversial figure because he was not an establishment type person Right and, in particular, this organization.
- 2:03Mark Graham: it was actually called Delta Force, I think it was called and not to be confused with the Delta Force the fast action alert team, and this was, I think, in the 70s, and they had a mandate to explore the human potential movement, yes, which was blossoming in California.
- 2:21Mark Graham: You know, hotbeds of activity included Esalen, for example, and people like Barbara Marks, hubbard, and certainly psychedelics was a big part of that, but also neuro-linguistic programming and NLP and, as Frank has explained to me, that at one point he had given NLP courses in the Pentagon for some of the upper brass and that a congressperson caught wind of this and required him to then give complementary courses to congresspeople, the idea being you know, this is kind of getting into this whole question of, like you know, asymmetric warfare that we don't want to have.
- 3:00Mark Graham: You know, we want to have both sides, if you will, of the equation have access to the same technology, the same tools.
- 3:07Brian "Ponch" Rivera: Interesting.
- 3:08Brian "Ponch" Rivera: Yeah, because it would have been contemporary to Boyd in his work.
- 3:10Brian "Ponch" Rivera: With Boyd's work, specifically, that same discussion occurred.
- 3:15Brian "Ponch" Rivera: You know, should we classify everything?
- 3:17Brian "Ponch" Rivera: So, like everything that we're versed in, that we read and we've read in the archives of the Rook Reader University in Quantico?
- 3:23Brian "Ponch" Rivera: You know Boyd was going to classify everything because, to your point, you know, you don't want the enemy to get this type of thinking in their hands.
- 3:30Mark Graham: You know, yeah, yeah, interesting.
- 3:31Mark Graham: So on the topic of go ahead, oh no, it's okay, we're just starting to get to know each other here a little bit.
- 3:36Mark Graham: You know, yeah, and I don't know, disclose, I'm a veteran as well.
- 3:45Brian "Ponch" Rivera: Oh, what branch.
- 3:46Mark Graham: US Air Force right Air Force yeah, air Force, just a short tour, like John Boyd, I guess so, except I wasn't a colonel, I was a senior airman when I left.
- 3:56Mark Graham: I did a short four-year tour from 79 to 83.
- 3:59Mark Graham: Okay, all right, well, I started out training to be a linguist and switched over to computers, so I spent a little more than three years in the Pentagon.
- 4:11Brian "Ponch" Rivera: We're really glad to meet you in our mutual connection of Mike and the Library of Alexandria is a topic that comes up a lot with the burning down and the archives weren't preserved and that's something that you're hard at work at Very much so Very much.
- 4:23Mark Graham: so.
- 4:23Mark Graham: You know, I was actually at the Computer History Museum out here in Silicon Valley on Sunday and the founder was there.
- 4:31Mark Graham: He'd been there for like 30 years and he says to me he goes you know, mark, you know, I hope you're taking this seriously.
- 4:38Mark Graham: I mean, this is really important stuff you guys are doing and then did kind of you know, catch me up a little short, like take stock of it, like said you know, this is our modern day.
- 4:49Mark Graham: You are the internet archive, you know, not me personally the whole organization of 120 some people.
- 4:55Mark Graham: You guys are the modern day library of Alexandria.
- 4:59Mark Graham: And so it is important that you take care of it.
- 5:01Mark Graham: Yeah, I'm sorry, yeah.
- 5:03Mark McGrath: No.
- 5:03Mark McGrath: So on that, mark Graham from the Internet Archives, I want to go back in time, thomas Jefferson talking about a virtual network that transmit ideas across time and space.
- 5:12Mark McGrath: We know about the.
- 5:15Mark McGrath: We brought up the Library of Alexandria.
- 5:16Mark McGrath: We also know that Thomas Jefferson had a lot to do with how it influences information flow and how we are excited about information flow flow and how we are excited about information flow.
- 5:24Mark McGrath: In fact, thomas Jefferson dreamt about a library that would provide unfettered access to the sum of human knowledge, right?
- 5:30Mark McGrath: So this continuation of that through your work, which I believe is what you're doing, is how do you ensure that human knowledge survives through time?
- 5:40Mark McGrath: So, starting there kind of walk us through how the archive came about.
- 5:44Mark McGrath: Just a light one.
- 5:45Mark Graham: Yeah, we're just going to start out of the shallow and kind of get deeper.
- 5:48Mark Graham: No well, first of all, we work to ensure or to increase the probability of.
- 5:54Mark Graham: You know, we tend to avoid absolutes.
- 5:57Mark Graham: So I'll just say that from the start, despite the fact that we have this mission statement, which is all about absolutes and superlatives, which is universal access to all knowledge.
- 6:08Mark Graham: So the origin of this venture goes back to a guy named Brewster Kahle, who started two companies on the same day, one, internet Archive, as a nonprofit and the other one, alexa Internet, as a for-profit and um, and he ended up selling alexa internet to amazon, jeff, jeff bezzo.
- 6:28Mark Graham: So then, uh, worked for jeff for a while, and alexa internet was one of the very first finding tools, if you will, for the emerging internet.
- 6:37Mark Graham: This is, we're talking, the very, very beginning days of the web and actually kind of the tail end of Gopher.
- 6:44Mark Graham: Before that, brewster had founded a company called Waze, white Area Information Service that he sold to AOL back before AOL had ventured into the web as we know it today.
- 6:54Mark Graham: So anyway, he ended up selling Alexa Intranet to Jeff Bezos for a fair amount of money and was then able to dedicate all of his attention to Internet Archive and, from the very early stage in the beginning, had this mission, this audacious idea of universal access to all knowledge.
- 7:15Mark Graham: And, in particular, what he did was, in effect, he hit record on the new media of the web.
- 7:23Mark Graham: So we've been working every day since then, for the last 28 plus years, getting better at that, getting better at recording more material that's made available via the web, and in that time, have expanded and diversified into a range of other media that human beings publish in.
- 7:45Mark Graham: Our mandate is to work toward acquiring and, if necessary, digitizing, preserving, organizing and making available the published works of humankind, and so that's what we get up every day and work toward.
- 8:03Mark Graham: I could go on.
- 8:04Brian "Ponch" Rivera: No, I mean, I use it a lot right.
- 8:07Brian "Ponch" Rivera: I use the archive a lot.
- 8:09Brian "Ponch" Rivera: I think that, well, maybe we should start too.
- 8:11Brian "Ponch" Rivera: We could talk about some of the detractors that are trying to stop what you're trying to do, because there is such an imperative to not only preserve that knowledge but also to make it available so more people can gain not just knowledge but also understanding.
- 8:25Mark Graham: Yeah, you, yeah, I guess we could go there if you wanted to.
- 8:27Mark Graham: But actually, why don't we set the groundwork a little bit around the media?
- 8:31Mark Graham: Because I mentioned that we certainly work to archive much of the public web more than 1 billion URLs a day.
- 8:39Mark Graham: So I think this year we'll probably exceed more than 1 trillion web pages archived actually, and a web page can be made up of lots of individual URLs.
- 8:50Mark Graham: So we're coming up on a bit of a landmark later in 2025.
- 8:55Mark Graham: And I can dig more into the web archiving work that we do.
- 9:00Mark Graham: But in addition to that, we archive books and academic papers and material on microfiche and microfilm and vinyl 78s, cds, television, in particular television news, radio, radio, news programs and other obscure media as it comes along.
- 9:26Brian "Ponch" Rivera: I love.
- 9:27Brian "Ponch" Rivera: The CIA reading room is one of my favorite places.
- 9:30Brian "Ponch" Rivera: I also like that you have all the DTIC, so a lot of the white papers that we use are in there.
- 9:37Mark Graham: Yeah, defense Technical Information Center, I think.
- 9:40Mark Graham: Dti service service.
- 9:42Mark Graham: Yeah, actually, you know, my favorite, okay, is FBIS.
- 9:48Mark Graham: Do you know that one?
- 9:49Mark Graham: No, oh, my God, okay, this is going to blow your mind.
- 9:51Mark Graham: So FBIS is the Foreign Broadcast Information Service, and this was an effort that started well, in some ways, it started about 100 years ago and it took on different names and different forms over time.
- 10:04Mark Graham: For many decades it was a collaboration between the US and Britain they had their own version of this and for many decades it was a division of the CIA, an unclassified activity of the CIA.
- 10:17Mark Graham: Actually, a version of it existed before the CIA was formed too.
- 10:21Mark Graham: What it did was it employed at one point, more than a thousand analysts around the world to pay attention, to collect, scan, synthesize publicly accessible news and then send it back to Washington, where it'd be put together and then turned into a variety of kinds of reports.
- 10:41Mark Graham: So, foreign broadcast information service, it was the broadcast from around the world on radio, on television, on newspapers, et cetera, and so, therefore, policy people and others in Washington, they could have some sort of an idea of, in fact, what the world was thinking, what people were being told was a reality in different languages and in different geographies.
- 11:10Mark Graham: I did a lot of work.
- 11:11Mark Graham: When I got out of the Air Force in 83, I shifted my attention to the nuclear war problem and was active in a variety of nuclear war-related activities.
- 11:22Mark Graham: I created a computer network for activists called PeaceNet and I got involved in a lot of US-Soviet citizen diplomacy work, which included traveling to the Soviet Union and bringing American-published books there, for example, and connecting with other people that were working with at that point, the emerging internet, and helping to establish and then operate an email service, the first commercial email service into and out of the Soviet Union.
- 11:53Mark Graham: So in any ways I was very interested in what was happening in the Soviet Union in the 80s and one of the primary ways that I could do that was by getting access to these FBIS reports that my government had the good sense to pull together.
- 12:07Mark Graham: That gave us an insight into what was happening there.
- 12:11Brian "Ponch" Rivera: So it gave more perspectives to help with the Yep.
- 12:14Brian "Ponch" Rivera: Yeah, I have to tell you too, I am the one, the other, I guess my favorite collection is you know, I'm a oh my God yeah.
- 12:23Mark Graham: Grateful Dead.
- 12:24Brian "Ponch" Rivera: Collection on archive Rock and roll.
- 12:26Brian "Ponch" Rivera: Oh, it's unbelievable.
- 12:27Mark Graham: I think we have, I don't know I almost tracked 13,000, maybe something like more than 10,000 concerts there.
- 12:33Mark Graham: Yeah.
- 12:34Mark Graham: The story behind that was, of course, as you know, they're from San Fran, right?
- 12:37Mark Graham: Oh yeah.
- 12:38Mark Graham: But the Grateful Dead community, where some may argue, are the creators, the originators, progenitors of this idea of sharing a sharing economy.
- 12:49Mark Graham: And, in particular, there was the people.
- 12:51Mark Graham: Tapers would create tapes I think at that point it was mostly cassette tapes, physical cassette tapes, cassettes and dats and dats, yeah.
- 13:00Mark Graham: And then they would pass them around, they would share them, and apparently the band was fine with this as long as no one made any money on it.
- 13:07Mark Graham: They were like, you know, yeah, like share and share alike, and so they did this for a long time.
- 13:12Mark Graham: And then, at one point, you know, brewster offered them unlimited storage, forever for free, and one of the main people was like I don't believe you, I don't know.
- 13:22Mark Graham: No, I appreciate, try me.
- 13:24Mark Graham: Just, you know, whatever.
- 13:25Mark Graham: And they did and like, as you're experiencing, it works.
- 13:29Mark Graham: That's unreal.
- 13:30Brian "Ponch" Rivera: So today, now, decades later, yeah, it's probably still that, one of the one of the things we're most well known for within certain well, I can, you know, I'm a, I'm high school, 94 right, but I I can remember filling shoe boxes full of bootlegs and then like passing them around and trading them, and then you could only get the tapes and then you would, you would, you would you know record, uh, and like the dual cassette decks or whatever yeah, side by side.
- 13:55Brian "Ponch" Rivera: I had one of those yeah, and then when I look at that collection, I just say this is what.
- 13:59Brian "Ponch" Rivera: What I would have imagined would have been like star trek level in my own life when I came to the grateful dead libraries.
- 14:05Mark Graham: Absolutely it's unbelievable passion.
- 14:08Mark Graham: You know, yeah, that's, that's great.
- 14:11Mark Graham: So, yeah, we got a lot of that.
- 14:12Mark Graham: We have a lot of not just the grateful dead.
- 14:13Mark Graham: We have a very large collection of of live concerts and these these are concerts that have been uploaded by band members.
- 14:21Mark Graham: What, what are some other things?
- 14:23Brian "Ponch" Rivera: like nuggets that, like you know, you talk about fbis, we talk about cia reading room dtic grateful dead like what are some other?
- 14:30Brian "Ponch" Rivera: Real treasure troves, because when you do go on the archive, if you don't have a map, I mean it is a lot of stuff.
- 14:37Brian "Ponch" Rivera: So how, what are some other like just amazing things you would guide people to and say, hey, go start here and then play around with this you know.
- 14:45Mark Graham: I mean, obviously people are drawn for their own passions, but things like well, knitting you know, plans for knitting is quite extensive, or my mom's going to love that.
- 14:57Brian "Ponch" Rivera: My mom's in the knitting cult.
- 14:59Mark Graham: Yeah, okay, well, you know you might lose her because she just may fall into that and never, never come out.
- 15:05Brian "Ponch" Rivera: Is that a separate, standalone thing, like the dead or the CIA?
- 15:08Mark Graham: Well, I mean it's their collection we organize.
- 15:11Mark Graham: So first of all it's at a very high level.
- 15:14Mark Graham: There's the kind of two sides.
- 15:16Mark Graham: There's the archiveorg, which is a series of collections, and then there's the Wayback Machine, and the Wayback Machine is like a special viewer, if you will, for the material that we archive from the public web.
- 15:29Mark Graham: But back onto the archiveorg side, I'd say another would be old-time radio and it's amazing.
- 15:34Mark Graham: It's like a certain age groups of people and I'll meet someone and I'll go oh my God, you guys do the old-time radio.
- 15:41Mark Graham: That's fantastic.
- 15:43Mark Graham: I listen to it every day.
- 15:44Brian "Ponch" Rivera: How are those things preserved like so, in the old days, say like the 30s, they had a broadcast going out of radio city, say I hear in manhattan, like what they would archive on tape or on record.
- 15:54Brian "Ponch" Rivera: Or how would they save that think?
- 15:56Mark Graham: I think it mostly that was on tape and then people then digitized it and then uploaded it to us.
- 16:02Mark Graham: We have someone on staff who's been focusing on ham radio, so shortwave radio, and it's very extensive and we're talking like the newsletters and the books and the reports and all of the ephemera related to that field.
- 17:18Mark Graham: And this person, that's all they do day in and day out is collect and digitize and make available that particular and curate that particular material for that the people that are interested in that that genre.
- 17:32Mark McGrath: Hey, mark, I'm wondering if you could share with us some of the challenges in preserving data like migration or emulation, and the reason for this is, I'm curious, from a perspective of business owners and preserving the past for the future, because if we misremember the past, then we can't improve the future.
- 17:58Mark Graham: So what are the challenges?
- 18:00Mark Graham: That, when people use our material, the mere act of them using the archive in all the different varieties of interfaces and emulations and media types etc.
- 18:13Mark Graham: That we work in, we learn about what's working, maybe what could be working better, maybe what isn't working so well, and that gives us an opportunity to continuously improve and to be able to stay current with what's available to people today.
- 18:29Mark Graham: And so a very specific example of that would be a media called Flash.
- 18:34Mark Graham: Right, so you know, flash was a media that was very popular on the Web as we know it about 25 years ago, and it was used in a variety of ways.
- 18:46Mark Graham: It could create like animated images, it could be used to stream a video, and many, many websites incorporated Flash.
- 18:54Mark Graham: In fact, some whole Web services were built entirely on Flash, and for a variety of reasons, in much of this, driven by the late Steve Jobs, flash ended up being killed, being abandoned by Adobe and the creator, and then the browsers over time dropped support for Flash.
- 19:13Mark Graham: So today, none of the main modern browsers will support the playback of a Flash object.
- 19:20Mark Graham: None of the main modern browsers will support the playback of a flash object.
- 19:22Mark Graham: So for some period of time, if there were flash objects on the Wayback Machine that we had archived, they would play back in browsers.
- 19:28Mark Graham: But now, if you're running Chrome or Firefox or Safari or something like that, you're not going to be able to play back a flash object.
- 19:39Mark Graham: So what we did was we found software, open source software.
- 19:41Mark Graham: What we did was we found software, open source software, an app called Ruffle, and you can think of this as like a middle layer or an emulation layer that sits between your browser and our backend archives that was archived in the Wayback Machine.
- 20:04Mark Graham: Then this Ruffle software will take the underlying code and it will, in real time, transform it and emulate a new kind of code in JavaScript that your browser will understand, and so you'll see what it was.
- 20:14Mark Graham: That you would have seen before if your browser had been supporting Flash.
- 20:19Mark Graham: Now is an evolution, it's not a, it's not a zero, and a one thing it says actually many, many features of flash is there was a very complex environment, and so we do not support all of the flash features.
- 20:33Mark Graham: We don't support action script level three, for example, yet and um, but then again, two, three years ago we didn't support any of it, and now we support a lot of it.
- 20:42Mark Graham: So there are millions and millions of images and animations and other interactions that we now support through the Wayback Machine.
- 20:51Mark McGrath: I wonder if you can help with an analogy here.
- 20:52Mark McGrath: So we have a 78, we have a record, we get this vinyl thing and we don't have a record player.
- 20:58Mark McGrath: Right, we don't have a way to play it, a means to play it.
- 21:00Mark McGrath: What Ruffle does is it allows you to play it, but it may not play all the features that are available on that vinyl, maybe.
- 21:07Mark McGrath: So this is important, I think.
- 21:09Mark McGrath: So what's happening is maybe the HTML or the scripts that are getting migrated over and over don't come with the way to decode them.
- 21:17Mark McGrath: That decoding secret is the Ruffle.
- 21:19Mark McGrath: It's another platform that kind of emulates how they were played in the past.
- 21:24Mark McGrath: And can you talk about the dangers of losing something like Ruffle?
- 21:28Mark McGrath: What would happen 15, 20 years down the road if we lost that capability?
- 21:33Mark Graham: Well, I mean one might lose the ability to interpret that material.
- 21:37Mark Graham: But a couple of things.
- 21:38Mark Graham: First of all, ruffle is open source, so it's not like some corporation is going to take away one's ability to run it anymore.
- 21:46Mark Graham: So it's out there.
- 21:47Mark Graham: Much of the software that we use at the Internet Archive is open source.
- 21:50Mark Graham: We contribute back to the open source community by improving the code and in other ways.
- 21:57Mark Graham: So there is that.
- 21:58Mark Graham: But the other thing is just because you can't read something doesn't mean that the preservation of it isn't useful, because maybe sometime in the future you will be able to read something, right, I mean?
- 22:14Mark Graham: I'll give an example.
- 22:15Mark Graham: The island of Rapa Nui, aka Easter Island, had a script and it's referred to as the Rongo Rongo script that was written onto these wooden boards.
- 22:26Mark Graham: It's still debated to this day whether the script was created pre or post Europeans visiting the island, european visiting Europeans visiting the island.
- 22:40Mark Graham: There's been some recent radiocarbon dating of these particular pieces of boards that suggest that some of them go back to the 1500s, so it was before the original 1722 of the European.
- 22:49Mark Graham: In any case, to this day there's an ongoing debate in academia about how to interpret the Rongo-Rongo scripts, and there's only about, I think, 28 or so of these pieces of wood still in existence, and so is.
- 23:03Mark Graham: So I'll just ask you just the fact, the fact that we can't yet definitively understand what these glyphs mean, right?
- 23:10Mark Graham: Does that take away the value of retaining these pieces of wood so that we can study them and maybe someday figuring it out?
- 23:17Mark Graham: Well, of course not we may get the.
- 23:20Mark McGrath: We may interpret it incorrectly in the future.
- 23:22Mark McGrath: That's what I I'm getting at.
- 23:23Mark McGrath: We don't know right, so that context that comes with it is important, and preserving context is important, right.
- 23:29Mark Graham: Preserving context is vitally important, and so the couple of words that really drive a lot of our work here one would be context, as you noted, and another would be provenance.
- 23:41Mark Graham: Maybe it's a particular special case of context that has to do with the origin of something.
- 23:47Mark Graham: So, for example, when we archive a URL, we record material like the time and the date that object was gotten, where it was gotten from, the IP number maybe of the machine that actually accessed it, the particular kind of software that was used when we made that archive, et cetera, and then we present that to people.
- 24:13Mark Graham: There's a link on the top right of every Wayback Machine playback page that says about this capture, and if you click on that link you can go back in time and you can see every page element on that web page.
- 24:25Mark Graham: Because, remember once again, web pages are made up of lots of individual little pieces of material image files, javascript, css, html, maybe a video object, et cetera.
- 24:38Mark Graham: And so you know, as an archivist working for a library, we care a great deal about a provenance, such that we can provide that particular context to readers in now and into the future.
- 24:53Mark Graham: Can you give us some context as to the rate that, I hate to say information, but part of it is a matter of definitions.
- 25:21Mark Graham: We have a lot of physics, for example, high-energy physics materials, cern and other entities, or, say, telescopes, climate change, data from satellites, et cetera, to produce massive exabytes worth of material.
- 25:43Mark Graham: That's just like, maybe, in some cases, raw data.
- 25:46Mark Graham: So, yeah, I don't know that I can really be an expert witness to that particular case, but I will say that, generally speaking, a couple of trends here, and one is that before the digital world we had mostly paper and, yeah, there was some television, there was some television, there was some radio, but paper has had been for a long time the main ways that people preserved knowledge and shared knowledge, and that was very useful for a variety of reasons.
- 26:11Mark Graham: First of all, paper tends to last a long time and also people tend to produce multiple copies of the things, multiple copies of the books, and so they physically spread them around.
- 26:22Mark Graham: They exist in most libraries, despite the fact that the history of libraries is one of destruction.
- 26:28Mark Graham: In fact, I'm going to reference your audience to a book called the Library A Fragile History.
- 26:34Mark Graham: It's a relatively recent book.
- 26:36Mark Graham: It's a tour de force of the history of libraries, which is pretty much a story of libraries being destroyed by churches and by governments.
- 26:47Mark Graham: I think now you might expand that to include corporations and lawyers.
- 26:52Mark Graham: Actually, despite the fact that libraries tend to not necessarily live forever, in many, many cases, you'd have a lot of libraries and a lot of books.
- 27:02Mark Graham: Generally speaking, the preservation of printed material is pretty good.
- 27:07Mark Graham: Now, when we moved over to the digital world and the advent of web servers and the like, things shifted a bit.
- 27:15Mark Graham: Instead of having maybe 1,000 or 10,000 or more physical books, Instead of having maybe you know, a thousand or ten thousand or more physical books, you'd have maybe like one web server living on one machine in one building run by one company.
- 27:30Mark Graham: And over the last few decades, many, many things have happened Companies have gone out of business, buildings have burned down, machines have failed and any one of a number of variety of threats that have caused the only copy of certain digital material to go away.
- 27:42Mark Graham: So I'll reference, for example, there was a recent Pew Research study that looked at a collection of web pages from 2013.
- 27:52Mark Graham: And this study was done in 2024.
- 27:55Mark Graham: And it found that for that particular collection of web pages from 2013, and it found that for that particular collection of webpages from 2013,.
- 28:24Mark Graham: Fully 38% of them were no longer available on the live web a little more than a month into this new presidential administration, and we've been involved in a project at the Internet Archive since 2008, called the End of Term Archive, where we work in collaboration with a number of other organizations.
- 28:35Mark Graham: Historically, we've done this with the Library of Congress, the National Archives and Records Administration, us Government Pub, as we can.
- 28:42Mark Graham: So, once again, we've been doing this since 2008, every four years, and so we started this project.
- 29:08Mark Graham: We do it tend to do it now in three phases before the election, after the election and then after the inauguration.
- 29:15Mark Graham: So we're now in phase three.
- 29:17Mark Graham: We're now working on the after inauguration archive, where we had identified tens of thousands of US government websites, and we go deep on them and we attempt to get as much of the material from those websites archived as we can.
- 29:32Mark Graham: Well, we've seen whole websites taken offline.
- 30:04Mark Graham: For example, just a few days ago, the that particular part of the Department of Education website.
- 30:06Mark Graham: We have archived and are now available in the Wayback machine, so you could type in a URL and you could see what was there just a week or two ago.
- 30:16Mark McGrath: So, mark, I'm curious who should be responsible for preserving websites.
- 30:19Mark McGrath: Is it the Internet Archive?
- 30:21Mark McGrath: Or I mean, what happens if you're not around.
- 30:24Mark Graham: I'm from California.
- 30:25Mark Graham: We try to avoid the word should.
- 30:26Mark Graham: Okay, okay, that's what my therapist told me.
- 30:29Mark Graham: No, there are no shoulds.
- 30:30Mark Graham: I know that's a great question, so don't mean to make light of it.
- 30:33Mark Graham: Look it really.
- 30:35Mark Graham: Well, I find it curious.
- 30:39Mark Graham: I'll just say I find it curious that this little, relatively small nonprofit out here in California does punch above our weight, if you will, and has historically and to this day taken on such a disproportionately large role in this seemingly really important effort to try to preserve the published works of humankind and make them available.
- 31:08Mark Graham: But be that as it may, here we are, be here now, right, and so that's what we're doing Now.
- 31:14Mark Graham: There are many other efforts.
- 31:15Mark Graham: So, for example, in America there's the National Archives, the National Archives and Records Administration and the Library of Congress.
- 31:21Mark Graham: They have efforts that they've been developing for a long time to archive much of this material.
- 31:28Mark Graham: For a long time the Library of Congress actually contracted with us so they outsourced much of their web archiving to the Internet Archive, the Internet Archive.
- 31:37Mark Graham: We're a nonprofit and we survive through the kindness of others donations by more than 150,000 individuals last year.
- 31:46Mark Graham: But in addition we have a program-related business activity called Archive-It, and Archive-It provides our web archiving services to schools and colleges, universities, museums, libraries and governments.
- 32:00Mark Graham: So many governments around the world contract with us to do web archiving on their behalf.
- 32:06Mark Graham: I already mentioned, for example, the Library Innovation Lab at Harvard.
- 32:12Mark Graham: They took on an effort to try to archive data sets from the US government and they archived something like 300,000 data sets.
- 32:20Mark Graham: So it's a big community of people, individuals, volunteers, organizations like ours, in some cases governments, government organizations and others that take on this work.
- 32:33Brian "Ponch" Rivera: We're talking about paper to digital, and I imagine there's a lot of collections yet to be archived.
- 32:39Brian "Ponch" Rivera: Potch and I are intimately familiar with one that we want to get archived and I want to help you with that, by the way.
- 32:46Mark Graham: So yes, we're going to follow up on this.
- 32:48Brian "Ponch" Rivera: We'll talk offline on that because it's critical.
- 32:50Brian "Ponch" Rivera: It's really critical to the work.
- 32:52Brian "Ponch" Rivera: But on that note, I mean of what's been digitized.
- 32:57Brian "Ponch" Rivera: Have we even put a dent in the stuff that's been collected over hundreds, maybe a thousand years?
- 33:05Mark Graham: we put a dent, um, but there's a lot of work to be done by.
- 33:08Mark Graham: By some accounts there's been more than 120 million books um produced so far.
- 33:14Mark Graham: By the way, um and uh, we've digitized maybe somewhere close to eight million of them.
- 33:20Mark Graham: Google has digitized um, many more than that, maybe a number north of 40 million.
- 33:27Mark Graham: I've heard as high as 40 million, but we're maybe number two, not really sure.
- 33:33Mark Graham: A lot of work to be done there.
- 33:36Mark Graham: There's a particular collection that I just started working on.
- 33:39Mark Graham: You guys might be interested in this one.
- 33:41Mark Graham: The Pentagon had ran a news clipping service for a long time called Current News, and Current News would come out twice a day once at 7.30 in the morning.
- 33:52Mark Graham: It was about 10 pages long, and one at 11.30 in the morning called the Main Edition.
- 33:56Mark Graham: It tended to be 20 pages or longer and it was physically produced in the Pentagon by people who would come in at their early hours.
- 34:06Mark Graham: They'd start arriving at about 2 am and then trucks would start delivering newspapers from all over the country and they would go through these newspapers and magazines and find articles related to military issues.
- 34:19Mark Graham: Find articles related to military issues, literally cut them with a knife and then paste them up on eight and a half by 11 inch pieces of cardboard or something and then photocopy them and then staple them together and distribute.
- 34:32Mark Graham: I think they distributed 4,000 a day throughout the building.
- 34:36Mark Graham: I collected close to 1,000 of these that I have physically with me now that I've just begun the effort to digitize and this particular collection of material.
- 34:46Mark Graham: Now maybe some of the individual stories, although I'm not sure about some of them because they're from the early 80s.
- 34:53Mark Graham: Maybe some of the individual stories exist in archives from the newspapers they were originally printed at right.
- 34:59Mark Graham: Maybe Many of them probably don't, because they never digitized those newspaper archives.
- 35:05Mark Graham: However, what I guarantee you doesn't exist is the curation of this material in the form of these 10 or 20-page compilations that were published every day.
- 35:18Mark Graham: And why is this important?
- 35:20Mark Graham: Because this particular collection of curated material was what our nation's military leaders were in many cases reading on a daily basis, and so it was helping to inform their worldview, and that, just it, fascinates me.
- 35:36Mark Graham: Yeah.
- 35:36Brian "Ponch" Rivera: Yeah, that's really interesting.
- 35:38Brian "Ponch" Rivera: What are the to your knowledge?
- 35:41Brian "Ponch" Rivera: What are the mother loads out there that need to be digitized, that have not yet, that are in critical maybe like critical state or in danger, that you know that?
- 36:02Mark Graham: Yeah, you know, like for like, for example, I know, I know that all the presidents that have their own libraries and things like that do those things get digitized automatically or not?
- 36:05Mark Graham: Like, not, no, not not necessarily like.
- 36:06Mark Graham: I could probably attempt to give you a good answer there, but I would have to say that at times my eyes have glazed over when I've reviewed catalogs of materials, yeah, that have yet to be digitized.
- 36:12Mark Graham: And uh, uh, you know, there was, for example, just one we we recently become aware of a collection of material at I think it's at Wright-Patterson Air Force Base a very large collection of Soviet-era military books and journals, and I doubt that that material has ever been digitized, that the Soviets have not done it, and if they have, it's not available to the West, and so that is a particular collection of hundreds of thousands of individual items for example.
- 36:46Mark Graham: That's just one that comes to mind.
- 36:49Mark McGrath: I do want to talk about John Boyd's archives here in a second, but before we do that question about how are you prepping everything for AI artificial intelligence what are you doing to prep the landscape for that?
- 37:01Mark Graham: Yeah, well, you know, we I'd say we generally think that we should let the computers read I'll just speak for myself here now and I think you know that AI is like a bicycle for the mind and that we as humans and you know the animals that we share this world with we frankly need all the help we can get.
- 37:23Mark Graham: We've got a, we got a bunch of big problems that that we're facing.
- 37:24Mark Graham: I've just list four of them pandemic, either intentional, through you know um attack, biological attacks, maybe climate change, nuclear war and then the pollution of our information ecosystem with mis and dis, dis, disinformation, so and, and these, they, these interoperate with each other, especially the last one with the other three.
- 37:44Mark Graham: So I think, generally speaking, we've got some big problems and we need all the help that we can get, and I think that AI holds promise to be able to help us with some of these problems, especially when you start adding things like quantum computing to the next, which seems to be potentially right around the corner New announcements out of Microsoft this last week but at the same time, at least within the United States, there are some significant constraints in the law, and so the ability for AI companies, ai technologies, maybe even open source AI projects to be able to access material and use material either training or some sort of a rag process on the back end is constrained by a lack of clarity in what is allowed.
- 38:35Mark Graham: That is currently playing out in the course.
- 38:37Mark Graham: So, instead of having you know a clear understanding of what can be done and what can't be done, it's being left to the courts to try to sort this stuff out.
- 38:45Mark Graham: And so many, many individuals and organizations are being cautious.
- 38:49Mark Graham: We are being cautious at the Internet Archive material, some of which we haven't mentioned in this conversation.
- 38:55Mark Graham: Certain collections of material that we consider by all accounts to be open and so that material that is open may be used, may be being used by some for a variety of applications, including AI training, but other material that is not generally open is no, not.
- 39:23Mark Graham: We have a term of use.
- 39:24Mark Graham: We remind people of that and we encourage people to use our library responsibly.
- 39:32Mark McGrath: Great, I appreciate that.
- 39:33Mark McGrath: So here's what I want to do.
- 39:34Mark McGrath: Mark and I are going to talk about why the archives John Boyd's archives are so important to us and kind of frame what's it like to go in there and why it's important that we have access to them and why we see the OODA loop as being different than the majority of folks that never been in there.
- 39:49Mark McGrath: So, Moose, when you and I go on there, we open a book and there's John Boyd's notes, marginalia all over the.
- 39:56Mark McGrath: You know all over everything.
- 39:57Mark McGrath: So we know the books that John Boyd read, but it's not enough to just download those books and have access to books that he read.
- 40:05Mark McGrath: It's more important to have access to the books that he physically wrote in right, and that's, I think, one of the more important things about having open access to John Boyd's work.
- 40:18Mark McGrath: Is that, Second Moose, what else I mean?
- 40:20Mark McGrath: We know it's open.
- 40:21Mark McGrath: We know that Mary Boyd has preserved this for everybody to have access to, but not everybody has access to it.
- 40:28Mark McGrath: You have to drive to Quantico, you have to get on base, you have to do these things to get in there.
- 40:31Mark McGrath: But any other thoughts as to frame a?
- 40:33Brian "Ponch" Rivera: picture for Mark.
- 40:34Mark McGrath: Graham.
- 40:35Brian "Ponch" Rivera: Yeah, I mean I tell you, mark, the take that we have on Boyd, which we know from his acolytes that worked with him directly, his daughter and others.
- 40:43Brian "Ponch" Rivera: I mean we're trying to advance and develop the authentic Boyd that was him, not the one that's commonly misunderstood and reduced to what most people know him for in there, and you're reading in his own writing, for example, his copy of Clausewitz on war, which is still a very definitive book for training officers in an advanced level.
- 41:07Brian "Ponch" Rivera: But you read the marginalia.
- 41:09Brian "Ponch" Rivera: You know John Boyd is absolutely tearing it to shreds, page by page by page by page, and to have all that marginalia saved in an archive.
- 41:18Brian "Ponch" Rivera: You know the book in the archive box.
- 41:20Brian "Ponch" Rivera: You know those are the sorts of things that people don't understand about Boyd and those are the things that kind of disconnected from his broader work on complexity.
- 41:30Mark Graham: We are here to help.
- 41:30Mark Graham: Absolutely.
- 41:31Mark Graham: Let's roll up our sleeves and just get this done.
- 41:34Mark Graham: We worked with another figure, daniel Ellsberg A US Marine, I might point out.
- 41:41Mark Graham: Indeed, he was, and Dan was a friend of the archive, and I consider Dan a friend.
- 41:51Mark Graham: He had a profound impact on my early life, and also Brewster Kahle, our founder, and so we actually had someone physically working in Dan's home for the better part of a year digitizing much of his material.
- 42:03Mark Graham: So, in that case, the Internet Archive or one of our friends actually, I think, maybe helped make that all possible, and we have a collection.
- 42:13Mark Graham: It's called archiveorg slash details, slash Daniel Ellsberg, and it has let me pull this up here.
- 42:21Mark Graham: It has 1,621 items in it, and these items tend to be books and papers and articles either written by Dan or written about Dan, and I can tell you that Dan, of course, passed away fairly recently.
- 42:39Mark Graham: Before he passed away, he gave me a call one day and he was like Mark, I'm trying to track down this copy of a particular book, but I think you had digitized, and it was in particular, because of the marginalia that was in the book that he was most interested in.
- 42:53Mark Graham: So we have experience doing this.
- 42:55Mark Graham: We can digitize books at scale depending upon the rights issues of the books, make them available to people.
- 43:04Mark Graham: Different books have different rights considerations associated with them, and so we operate within the constraints of law.
- 43:12Brian "Ponch" Rivera: So if a government agency say like CIA, they're like hey, we're going to work with the archive and we're going to digitize the CIA reading room.
- 43:18Brian "Ponch" Rivera: I mean, does the CIA with the archive and we're going to digitize the CIA reading room?
- 43:21Brian "Ponch" Rivera: I mean, does the CIA pay the archive?
- 43:24Brian "Ponch" Rivera: Or how does that work?
- 43:25Brian "Ponch" Rivera: Not yet.
- 43:29Mark Graham: The CIA reading room is what I think a collection is called an archive.
- 43:32Mark Graham: That work has mostly been done by, to the best of my knowledge, correct me if I'm wrong here, guys, but the National Security Archive out of Georgetown University, I think, is one.
- 43:48Mark Graham: Is it Georgetown or George Washington?
- 43:51Mark Graham: So the National Security Archive is one and they've done heavy lifting on this.
- 43:56Mark Graham: But they file FOIA requests, et cetera.
- 43:58Mark Graham: They file FOIA requests, et cetera.
- 43:59Mark Graham: But there's also, as an individual, there's a couple of individuals who have just made it their life's passion to work for years and years and years acquiring materials, digitizing them and uploading them to the Intran Archive.
- 44:13Mark Graham: So hundreds and hundreds of thousands of documents, in fact, maybe even millions of documents.
- 44:20Brian "Ponch" Rivera: Does it work like when an archive gets sealed?
- 44:24Brian "Ponch" Rivera: For example, I went to Marquette University and one of our more notorious alums, both undergrad and law school, is Joe McCarthy, and I had a professor at Marquette that wanted to go and do research on McCarthy's transcripts from college.
- 44:37Brian "Ponch" Rivera: What classes did he take and what were his grades?
- 44:39Brian "Ponch" Rivera: Research on mccarthy's, you know, like his transcripts from college, like what, what classes did he take and what were his grades?
- 44:43Brian "Ponch" Rivera: And like the, the joe mccarthy archive is sealed till, like you know, 2150 or whatever it is, I don't even know.
- 44:49Brian "Ponch" Rivera: But like, how did, how did the?
- 44:51Brian "Ponch" Rivera: What are the usual protocols for things like that?
- 44:54Brian "Ponch" Rivera: Are they waiting for people to die?
- 44:55Brian "Ponch" Rivera: Are they waiting for?
- 44:57Mark Graham: uh, you know, like, I think I think, generally speaking and this is actually one of the principles that we follow at the Internet Archive, which is that people who are rights holders can choose.
- 45:08Mark Graham: So if, for example, if there's maybe some material that's been archived in the Wayback Machine, maybe somebody's blog say, for example, from 10 years ago, and for whatever reason they say, I don't want that in the Wayback Machine.
- 45:23Mark Graham: I deleted my blog a long time ago and I'd like you to take it out of the Wayback Machine.
- 45:27Mark Graham: So we first check to make sure that they are who they say they are, take some reasonable effort.
- 45:33Mark Graham: There's some appropriate due diligence, and then I generally honor those requests.
- 45:37Mark Graham: So I'd say it's really about the rights holders, anything else and and try to do the right thing by people as well.
- 45:45Brian "Ponch" Rivera: I can't really speak to that particular mccarthy archive per se no, it's just fascinating because it did sound like it was in, like it did belong to, like his wife or his estate or something like that.
- 45:55Brian "Ponch" Rivera: Yeah, and then maybe they had then, but I guess, a lot of colleges though, because you mentioned georgetown, yeah I feel like marquette was one of these that are sanctioned by the national archives to have those sorts of facilities that could be sealed and not opened um to secure those kind of documents yeah, there's also a program at columbia university that dealing deals with declassified documents in particular, quite a quite a significant program.
- 46:20Mark Graham: You you might want to look into that.
- 46:22Brian "Ponch" Rivera: Are there ever archives that you wanted, that you sued for, or like to put it like a FOIA request, or?
- 46:29Mark Graham: that kind of stuff.
- 46:30Mark Graham: I mean, I've done FOIA requests and we certainly have people that do FOIA requests, that upload material from those results to the interim archive, but no, that's not really something that we spend a lot of time on.
- 46:41Brian "Ponch" Rivera: Yeah, just curious, yeah Interesting.
- 46:44Brian "Ponch" Rivera: I mean it's like the Wayback Machine is like a time machine, it is.
- 46:50Mark Graham: Absolutely.
- 46:50Mark Graham: It's a time machine for the web, you know, because the web has no version control system and because we've as I've already, you know, shared the web is fairly fragile and it's not necessarily getting any better right With the rise of platforms.
- 47:06Mark Graham: In many cases, you have power consolidated down into a small number of very large organizations that can change their policies on a dime, or, as we've seen most recently with the US government.
- 47:18Mark Graham: Now USAID's entire website was taken offline, and not just UID's website, but and I don't have the exact numbers, but I think there were something like 29 different channels on YouTube managed by the USAID, and all of those videos were taken off of YouTube.
- 47:34Brian "Ponch" Rivera: Wow, wow.
- 47:36Mark Graham: So the internet never forgets.
- 47:37Brian "Ponch" Rivera: Sometimes it does.
- 47:38Mark Graham: Well, sometimes it does, but I think we have preserved most of those videos in the wayback machine just I mean, what technical question do do ai we could punch it out ais do?
- 47:50Brian "Ponch" Rivera: are there like api, abilities that ais can look into?
- 47:53Brian "Ponch" Rivera: Uh the archive or or no.
- 47:56Mark Graham: once again, the internet archive makes materials available for a variety of different audiences, and certain open material may be available and used by some of the AI companies.
- 48:09Mark Graham: That's really orthogonal to the question of how it's done.
- 48:13Mark Graham: And there are a variety of interfaces to the various services of the Internet Archive, including web-based iOS and Android apps, browser extensions and, of course, apis Fascinating.
- 48:28Brian "Ponch" Rivera: Well, we want to respect your time.
- 48:30Brian "Ponch" Rivera: We know we're coming up on your hard stop and we'd love to have you back.
- 48:33Brian "Ponch" Rivera: We want to take some time to ask a few questions.
- 48:35Brian "Ponch" Rivera: Offline Sounds good, mark.
- 48:37Brian "Ponch" Rivera: We really appreciate you coming on the show Been a pleasure.
- 48:40Brian "Ponch" Rivera: And sharing us how our digital orientation is expanding and helping us sense, make and make better decisions Right on.
- 48:46Brian "Ponch" Rivera: So all right, thanks.
- 48:48Brian "Ponch" Rivera: One day at a time, all right, that's it.
- 48:50Brian "Ponch" Rivera: Thanks, mark.