Latest / The Biorevolution Podcast / Living and Learning in the Metaverse: Stepping Inside Scientific Data

Transcript

Living and Learning in the Metaverse: Stepping Inside Scientific Data · The Biorevolution Podcast

• 0:04
  Speaker 1: The Biorevolution podcast.
• 0:06
  Speaker 2: Your hosts?
• 0:07
  Speaker 1: Luise von Stechhoff
• 0:08
  Speaker 2: and Andreas Roichler.
• 0:11
  Speaker 2: Welcome to the BioreVolution Podcast, this time again from Berlin!
• 0:15
  Speaker 2: And I'm excited we have a guest who will be talking about living learning in the metaverse – stepping inside scientific data….
• 0:23
  Speaker 2: …I might try to translate it.
• 0:26
  Speaker 2: We're moving from text N-to D into three D and less text maybe... But easy, as always we start with a quote.
• 0:35
  Speaker 1: We do and today I think it will quote one of the tech people who have not quoted yet because i'm seeing that recording all these friendly individuals to make our lives so much easier.
• 0:45
  Speaker 1: So Today It Will be Mark Zuckerberg Who seems quite invested in this idea of virtual spaces hanging out together in virtual spaces and moving parts of our life there.
• 0:56
  Speaker 1: And something that he said is the metaverse will be the successor to mobile internet, so this maybe a bit more broader understanding.
• 1:06
  Speaker 1: today we'll dive into what it could mean for science and scientific data visualization.
• 1:11
  Speaker 2: Along the way you might solve your problem if people take Metaverse as company versus a platform or an idea, whatever might coin for that and we'll get to it.
• 1:23
  Speaker 2: But first off Dr Andreas Büchles with us research lead in the cyber infrastructure of Network Science Center at Indiana University in Bloomington.
• 1:32
  Speaker 2: nobody knows where Bloomingt is but anyways its somewhat close to Chicago Illinois.
• 1:37
  Speaker 2: I think
• 1:38
  Speaker 0: That is pretty fair to say.
• 1:40
  Speaker 0: It's a four-hour drive, four and half hours with bad traffic.
• 1:43
  Speaker 2: Andreas Andy we might call him.
• 1:45
  Speaker 2: at the course of this podcast he played a key role in developing immersive tools for the NIH supported human reference athletes including the HRA organ gallery and The new HRA powers of ten And here has been thinking about what it might mean To live learn in the metaverse since well before the term became a technology buzzword.
• 2:06
  Speaker 2: right welcome.
• 2:07
  Speaker 0: Thank you very much for having me today.
• 2:09
  Speaker 0: This is really, really exciting!
• 2:11
  Speaker 0: I haven't been back to Berlin since i moved to the US so this was just a really nice... Really nice to be here.
• 2:16
  Speaker 0: and
• 2:16
  Speaker 2: your bewildered
• 2:18
  Speaker 0: Yeah, it's just a lot of new buildings.
• 2:20
  Speaker 0: A lot of New People.
• 2:21
  Speaker 0: is everything has kind-of changed when I lived here in the East and i didn't spend all that much time In this part of town.
• 2:28
  Speaker 0: so its discovery
• 2:30
  Speaker 2: for sure.
• 2:30
  Speaker 2: It makes us even more happy That you found your way to get Here.
• 2:35
  Speaker 2: You brought quote as well For the intro?
• 2:37
  Speaker 0: Yes!
• 2:38
  Speaker 0: Since we talked about The Metaverse And since it Is so prominent in the title This is a Quote by Neil Stephenson's novel Snow Crash Novel introduced the term metaverse and it also popularized a term avatar, which we still use everywhere.
• 2:54
  Speaker 0: So in The novel snow crash there is an environment of virtual platform called that the metaverse.
• 3:01
  Speaker 0: Which?
• 3:02
  Speaker 0: Is A three-dimensional Platform That people join with headsets much like you know the kind Of headset that We used today which I will show here.
• 3:10
  Speaker 0: In second And in There there is a constant tension between People's actual Living circumstances and their space in the metaverse.
• 3:20
  Speaker 0: And so, The protagonist of a novel who is ironically called Hero Protagonist that's actually his name.
• 3:26
  Speaker 0: You know lives on very small storage container In shipyard or something but he has a fixture of the meta-verse because He isn't founding member Of big club called Black Sun.
• 3:38
  Speaker 0: So A quote from this novel Is That why you can have nice Big House in the Metaverse but have to share a tiny apartment?
• 3:47
  Speaker 0: in reality, real estate acumen does not always extend across universes.
• 3:53
  Speaker 0: This is one of the central points of tension.
• 3:56
  Speaker 0: it's a cyberpunk novel.
• 3:57
  Speaker 0: and so with the difference or like the tension between individuals freedom and then the collective And The influence of tech Is very prominent there?
• 4:06
  Speaker 0: So that's where I thought this was pretty good quote about the metaverse.
• 4:11
  Speaker 2: Yeah amazing for intro.
• 4:12
  Speaker 2: you see i think we might like to start with clearing the terms that will be used also on this show today because I think a lot of people might be really confuzzled.
• 4:24
  Speaker 1: Yeah, including ourselves.
• 4:27
  Speaker 1: and it's interesting enough even though this buzzword is out there many don't have contact with metaverse virtual reality augmented reality extended reality mixed reality.
• 4:42
  Speaker 1: So maybe we could lay the land, what does all of that really mean?
• 4:46
  Speaker 0: Yeah it's funny you bring that up.
• 4:48
  Speaker 0: when I wrote my dissertation there was one of the first things i had to take care off early on in my prior work is... What do those things even mean?
• 4:56
  Speaker 0: ...I have to explain it tomorrow and dissertation committee as well.
• 4:59
  Speaker 0: Another thing thats challenging.
• 5:01
  Speaker 0: There are different companies having different terminologies.
• 5:04
  Speaker 0: so for example Apple when they, you know brought out the Apple vision pro.
• 5:09
  Speaker 0: They avoided using the terms VR a are all together.
• 5:13
  Speaker 0: They called it spatial computing and what we would usually call virtual reality?
• 5:18
  Speaker 0: They called immersive computing.
• 5:20
  Speaker 0: And what will call augmented reality?
• 5:21
  Speaker 0: The called semi-immersive computing.
• 5:24
  Speaker 0: so I also didn't call that ahead said they called at a special computer.
• 5:27
  Speaker 0: So there's a war of terminology going on.
• 5:29
  Speaker 0: their.
• 5:30
  Speaker 0: so extended reality is an umbrella term that encompasses all of these oval terms.
• 5:36
  Speaker 0: So I usually use the term extended reality or X are just to refer to all of this so-called interface modalities, virtual reality is where the entire world you see as digital.
• 5:50
  Speaker 0: think about it.
• 5:51
  Speaker 0: actually what we have in the snow crash novel were your wear one those bad boys and don't seen physical worlds anymore.
• 5:58
  Speaker 0: right
• 5:59
  Speaker 2: then its VR.
• 6:00
  Speaker 0: That's VR, virtual reality.
• 6:02
  Speaker 0: It is like the extreme end of the spectrum where everything you see is digital and then in the middle it's a large area mixed augmented reality And those terms are a little bit fuzzy but generally means that you can see a fair amount of physical content.
• 6:20
  Speaker 0: This can either be optical seethrough Where you actually see the actual world through screen.
• 6:26
  Speaker 0: Then there was display inside which overlays content, the Microsoft HoloLens that some of you might have used before is an example of that.
• 6:35
  Speaker 0: It's optical see-through.
• 6:37
  Speaker 0: and then there was digital augmented reality.
• 6:39
  Speaker 0: The MetaQuest three can actually do That.
• 6:41
  Speaker 0: Apple Vision Pro Can Do it too where You Have a video feed A high resolution High Frequency Video Feed Which Is Of Course Very Powerful Because It Means You Can Analyze The Video Feed In Real Time And Then Overlay Content On TopofThat.
• 6:56
  Speaker 0: So this is what those are.
• 6:57
  Speaker 0: Typically, it's just easiest to distinguish between VR where everything digital and AR were some of it as digital And most if that was the physical environment.
• 7:26
  Speaker 2: Where do we stand right now in use cases with XR as an umbrella term again?
• 7:44
  Speaker 0: on the verge of becoming big.
• 7:45
  Speaker 0: It's always around the corner, but it has not really quite broken through yet.
• 7:49
  Speaker 0: as far as use case go you already mentioned gaming.
• 7:52
  Speaker 0: entertainment i think is a potentially big one.
• 7:55
  Speaker 0: still virtual concerts just watching a movie and then suddenly I have...it feels like we have gigantic TV in front even though your wall is not big enough to actually hold that TV.
• 8:14
  Speaker 0: So something seemingly very simple, like watching a movie which you might as well do on any other device that you own.
• 8:21
  Speaker 0: it's quite engaging in this virtual environment.
• 8:25
  Speaker 0: The Apple Vision Pro allows you while sitting at the middle of Yosemite National Park in winter Which is kind-of silly.
• 8:32
  Speaker 0: but this is an actual valid use case.
• 8:34
  Speaker 0: changing the environments where you consume and already established media.
• 8:40
  Speaker 0: And then of course, scientific visualization.
• 8:43
  Speaker 0: Any kind of visualization that requires a three D and spatiality for that?
• 8:48
  Speaker 0: Of course there is pretty big use case to them.
• 8:50
  Speaker 0: off course these use cases get more and
• 8:54
  Speaker 2: coming from paper, right?
• 8:56
  Speaker 2: To science.
• 8:57
  Speaker 2: And that should be the focus of this show because we're in the Birolution podcast.
• 9:02
  Speaker 2: obviously so...
• 9:03
  Speaker 1: Yeah absolutely but I think other use cases are also super interesting for listeners and our viewers.
• 9:09
  Speaker 1: But i find really interesting something you just said.
• 9:12
  Speaker 1: it's always around the corner This technology.
• 9:15
  Speaker 1: So what was wondering?
• 9:16
  Speaker 1: where do you see the biggest hurdles?
• 9:18
  Speaker 1: at moment It is getting there a tool that is known, I would say.
• 9:26
  Speaker 1: probably in gaming communities, it's quite well known.
• 9:30
  Speaker 1: beyond that.
• 9:31
  Speaker 1: I think there are some use cases.
• 9:32
  Speaker 1: for example my husband is an architect.
• 9:34
  Speaker 1: they used to do virtual tours through their buildings with the clients.
• 9:38
  Speaker 1: so there are real world use case already but they're pretty specialized and most of us including myself have worn VR headset a couple times always like at a museum where you had immersive art exhibition or something in a real-world setting, let's put it like that.
• 9:57
  Speaker 1: And I think for this question what are the hurdles?
• 10:01
  Speaker 1: What do you actually need to have?
• 10:03
  Speaker 1: virtual reality?
• 10:04
  Speaker 1: because there is more than hardware right and also needs a virtual space to go into?
• 10:09
  Speaker 1: Right!
• 10:10
  Speaker 0: I would say they're all three big hurdles.
• 10:12
  Speaker 0: The first one might be easy to overcome but many people still see it as gimmick something fun to use but not really something that is serious.
• 10:23
  Speaker 0: So here, it's our job to provide user scenarios that are convincing and of course there does a pretty big effort in the data visualization community for example To go beyond that.
• 10:35
  Speaker 0: or this is a really cool demo too I actually want to use this every day Or very specific tasks And i wanna put these things on Forget about all other stuff That have been done.
• 10:46
  Speaker 0: I only do my task There.
• 10:48
  Speaker 0: so definitely one big hurdle.
• 10:51
  Speaker 0: The second one I would say is there isn't a need to get specialized hardware.
• 10:56
  Speaker 0: These things are much cheaper now than they used to be, but that's still about the price of our gaming console and as we all know prices for these things don't go down because it goes up with all components shortages.
• 11:09
  Speaker 0: so this is still a hurdle Now.
• 11:12
  Speaker 0: some universities or institutions might just be around in their budget But globally speaking.
• 11:20
  Speaker 0: And then the third one, which I think is a really significant.
• 11:23
  Speaker 0: One is The challenge of content creation.
• 11:26
  Speaker 0: for it now we have lived through kind of a democratization off video production and audio production over the past thirty years where It has really straightforward.
• 11:36
  Speaker 0: Now to create your own content all you need Is A camera computer?
• 11:41
  Speaker 0: Then there's so much software out.
• 11:42
  Speaker 0: There are So many tutorials on how To Make Your Own Videos that it is a viable business model now for people in almost every country.
• 11:51
  Speaker 0: With these things, we're definitely not there yet.
• 11:54
  Speaker 0: broadly speaking There's three ways of making content for VR headset.
• 11:58
  Speaker 0: one Is to do it on the web using a web browser.
• 12:02
  Speaker 0: One is to develop standalone application with a video game engine and then they are custom content creation pipelines For specific headsets.
• 12:12
  Speaker 0: none of them are easy.
• 12:14
  Speaker 0: What has been helpful?
• 12:15
  Speaker 0: there is the advent of LLMs for writing code faster, but it's still really complicated.
• 12:22
  Speaker 0: For our everyday life where we would be useful to think about training simulations with fire fighters as a pretty classic example... Surgery!
• 12:33
  Speaker 1: I think surgery is such an excellent example
• 12:36
  Speaker 0: of
• 12:37
  Speaker 1: practicing in virtual spaces.
• 12:40
  Speaker 0: The content creation needs to be bespoke And that is a big challenge to having this adopted.
• 12:46
  Speaker 0: So these three things, I think are pretty big hurdles there we have to overcome
• 12:50
  Speaker 1: before we move to the scientific data.
• 12:52
  Speaker 1: maybe one question where do you see?
• 12:53
  Speaker 1: because it was nice analogy?
• 12:55
  Speaker 1: I think for content creation in terms of audio video or content.
• 12:59
  Speaker 1: Do You See?
• 13:00
  Speaker 1: Because This Is More Like A Bottom Up Movement?
• 13:02
  Speaker 1: Right i mean These Are more Just People Creating Content and Either It's Successful Or Not But Its not Something coming from a bigger tech company or like, a bigger initiative.
• 13:12
  Speaker 1: For the virtual reality do you see such as bottom-up movement happening with lot of little virtual spaces emerging?
• 13:21
  Speaker 1: Or rather feel okay where someone is going Apple or Meta and giving us space that we all have to live in there?
• 13:29
  Speaker 0: I think every online ecosystem needs healthy amount of developers And so in our case, we developed for the Metastore which has been a pretty straightforward experience and I absolutely think there is a nice cottage industry of developers publishing.
• 13:47
  Speaker 0: It's probably not as lucrative as developing for the mobile market but all of the technical foundations are there.
• 13:54
  Speaker 0: you can use one of the established game engines to publish your games on.
• 13:57
  Speaker 0: that is absolutely doable!
• 13:59
  Speaker 0: The success of these online storefronts like Steam which was hugely successful I think it's quite promising for the VR market as well.
• 14:08
  Speaker 0: The question that developers has to ask themselves is how many people are going be able reach by supporting a specific platform?
• 14:16
  Speaker 0: And of course, the biggest reach in the world is with smartphones and so there was a lot of questions at.
• 14:22
  Speaker 0: developers have asked them when they start deploying their software but i would absolutely agree That It Is A Bottom Up Movement developer ecosystem, just like on any other platform.
• 14:55
  Speaker 2: Do you see any development?
• 14:57
  Speaker 2: I'm just thinking as somebody who makes sample-based music for instance.
• 15:04
  Speaker 2: we have around when you compose with sampled instruments For instance We had the MIDI protocol a hundred and twenty seven commands.
• 15:14
  Speaker 2: Everybody agrees to every industry player Every developer every scientist On the basis of this platform, everybody agrees to develop new stuff saying okay.
• 15:26
  Speaker 2: One hundred and twenty seven is enough.
• 15:29
  Speaker 2: if you play a note long shorts staccato with vibrato whatever it is You know then that's sufficient right?
• 15:36
  Speaker 2: And I could see like a framework Like you take three or four best game engines and If you want to go science approved Then you follow this path.
• 15:48
  Speaker 2: Is there any development into this direction?
• 15:50
  Speaker 2: Because what you say is, uh... There's a certain degree of anarchy which is sympathetic in a way because it's the core of the bottom-up movement.
• 16:00
  Speaker 2: Of course but on the other hand It's complicated when you have to do bulletproof stuff.
• 16:06
  Speaker 0: Yeah yeah that's a great point.
• 16:08
  Speaker 0: so there was lot like work going around standards.
• 16:12
  Speaker 0: I think one of the strongest ones we had were the OpenXR.
• 16:15
  Speaker 0: standard effectively allows you as a developer to deploy your software.
• 16:19
  Speaker 0: too many different headsets and many different input modalities.
• 16:24
  Speaker 0: You know where you can deploy it two headset with augmented reality in hand tracking or virtual reality was controller tracking.
• 16:32
  Speaker 0: does all of these different combinations?
• 16:34
  Speaker 0: so they are definitely standards that have been development for decades, and there are entities that actively develop an adjust the standards.
• 16:42
  Speaker 0: One of the most interesting standardizations, I think is going on around deploying VR or AR software through the web browser with WebXR.
• 16:51
  Speaker 0: It's an API that allows the browser to communicate with a hardware and do it in many different types of hardware.
• 16:59
  Speaker 0: like you have these headsets from all of those companies.
• 17:03
  Speaker 0: by having this shared API, this application programming interface they can talk to the browser.
• 17:10
  Speaker 0: So these are very, very technical standards that are being managed so that developers like me don't have to worry about them part.
• 17:18
  Speaker 0: Of course depending on the framework that I use i can still have a lot of low-level control but I don't Have too and you're going back To your point earlier what's the bottom up movement?
• 17:27
  Speaker 0: That is Very important.
• 17:29
  Speaker 0: most You know People who develop interesting And good software that many people want to Use Are not necessarily Good coders.
• 17:37
  Speaker 0: their strengths lie elsewhere.
• 17:39
  Speaker 0: It's kind of a parallel to in the early nineteen nineties when animation software became usable by designers without a mathematical or computer science background, computer scientists have many skills but they're not necessarily good animators.
• 17:54
  Speaker 0: so Toy Story for example was possible because three D software had gotten into point where people who were good at designing interesting spaces could use them and This is a similar kind of convergence that we see for VR development.
• 18:08
  Speaker 0: It's still too complicated, though I think for most folks to develop their own space.
• 18:14
  Speaker 1: I have to say listening to you... ...I can also why this scientific revolution of the R-spaces has not yet happened.
• 18:24
  Speaker 1: because there are so many open questions and points to solve right?
• 18:29
  Speaker 1: A lot things aren't as mature yet might think and they need to come together.
• 18:36
  Speaker 1: To be more accessible for the everyday man, yeah?
• 18:40
  Speaker 0: And it still has this very futuristic aspect of brief story.
• 18:44
  Speaker 0: one half years ago we went to Tokyo for a conference... ...and we had a day with at Sony.. ..and they have a spatial reality display section!
• 18:54
  Speaker 0: They just showed us some of their hardware a playground.
• 18:58
  Speaker 0: I mean, it was fantastic like screens that holographic screens.
• 19:03
  Speaker 0: they give you the illusion of looking at something in three D all with SDKs or software development kits.
• 19:09
  Speaker 0: so i could have taken out my laptop and develop an application for right there Like All Of The Hurdles had been removed.
• 19:15
  Speaker 0: but this is so futuristic And most people when see these don't immediately use case.
• 19:23
  Speaker 0: It's the job of people like us to convince.
• 19:25
  Speaker 0: People that hey, there is indeed a use case in their.
• 19:27
  Speaker 0: for you this is not just gimmick from future
• 19:30
  Speaker 1: which I think also nicely brings us to the question off using it for scientific visualization because i think they're all.
• 19:37
  Speaker 1: so what?
• 19:37
  Speaker 1: I feel maybe our imagination still too limited.
• 19:41
  Speaker 1: really thing about i think this practicing surgery example or may be looking at an organ examples kind of obvious, but I mean it goes way beyond that right?
• 19:51
  Speaker 1: And its not just having a space where you can see or explore things in three D. It's much more than that to access scientific data and virtual space.
• 20:02
  Speaker 1: potentially what changes from looking at our flat screen and clicking with the mouse?
• 20:07
  Speaker 1: two yeah stepping into the data.
• 20:10
  Speaker 0: well everything change is an I have to start by saying or in VR for that matter.
• 20:19
  Speaker 0: It's very easy to get into a mindset of, I just want to throw everything into three-dimensional space but it is not always helpful.
• 20:27
  Speaker 0: In general we say data already inherently spatial benefits greatly potentially from virtual reality display and as somebody who loves VR there value to simplicity.
• 20:41
  Speaker 0: Two D typically means simplicity.
• 20:43
  Speaker 0: There's a reason why visualizations like a bar graph or align graph, or scattergraph are so omnipresent because they allow us to reduce complexity to just the few dimensions and see patterns that way.
• 20:55
  Speaker 0: So where I think scientific visualization is really powerful Is if you show something?
• 21:01
  Speaker 0: That's already three d in its native dimensions that the user can scale up and down and you enrich it with additional two-dimensional graphics.
• 21:09
  Speaker 0: There's a really seminal paper from three by Doug Bowman, colleagues from University of Virginia.
• 21:14
  Speaker 0: I believe that i read during my dissertation about information rich virtual environments or IRVEs which means in the three dimensional environment.
• 21:24
  Speaker 0: but you enriched these three D objects in front of view with to dimensioning formation that displayed on up into dimensional panel.
• 21:32
  Speaker 0: this people was way ahead if its time.
• 21:35
  Speaker 0: last year I demoed one of our organ applications to.
• 21:41
  Speaker 0: I had no idea that she was a co-author, but we got to talking after the demo and.
• 21:45
  Speaker 0: That was very interesting like kind of closing the loop moment.
• 21:48
  Speaker 0: so i think the ability to bring two d content into three D has a lot really great potential?
• 22:09
  Speaker 1: both of spatial scale but also time-scale, and here I think this work you've done for the human reference atlas is super interesting.
• 22:17
  Speaker 1: For a user to understand this magnitude or scale that has cells, molecules, tissue, organism which on a graphic you put all those in an image put a tiny dot and or you have these comparisons is like, I don't know.
• 22:37
  Speaker 1: Like the way from the earth to the moon or something.
• 22:39
  Speaker 1: but that doesn't i think make it as explorable as really seeing with your own eyes right?
• 22:45
  Speaker 1: And navigating through it could maybe lead us a little bit through the work that you've done for Human Reference Atlas and how there You would feel like.
• 22:53
  Speaker 1: having this.
• 22:54
  Speaker 1: virtual tools makes them more explorable what users can take away
• 22:58
  Speaker 0: Absolutely.
• 22:59
  Speaker 0: There was an old movie called The Magic Voyage About a bunch of people that get shrunk and it's a plot point in quite a few TV shows That have explored this concept where if you scale somebody down And you put them into an otherwise everyday object Everything becomes different.
• 23:17
  Speaker 0: Yeah, and usually what stands out there is suddenly You see a lot more detail or conversely as a developer?
• 23:24
  Speaker 0: You need to show a lot More detail and that's actually already pretty challenging part oftentimes the data that you have Caught at one scale and so if you want the user to scale it up, You suddenly need to bring in additional details.
• 23:37
  Speaker 0: So The human reference Atlas To those that don't know is a comprehensive overview of the anatomical structures That make up our body across so far.
• 23:47
  Speaker 0: eighty-one three D organs these cell types that are located In this anatomica structures And the biomarkers that our genes or proteins or lipids dead characterize these cells.
• 23:59
  Speaker 1: And if I may add, the interesting point is that it's in healthy tissues right?
• 24:03
  Speaker 1: So we have a lot of information about disease but often we lack the reference standard and i think this is a great project to provide exactly that.
• 24:13
  Speaker 0: Yes!
• 24:14
  Speaker 0: We are trying to add disease data now one of the first challenges already there.
• 24:19
  Speaker 0: what exactly is healthy?
• 24:21
  Speaker 0: There is actually, there's not a lot of consensus on
• 24:23
  Speaker 2: one philosophical question.
• 24:24
  Speaker 2: Yeah?
• 24:25
  Speaker 0: On what normal health you know.
• 24:27
  Speaker 0: for some organs it might be pretty clear for others, not clear at all.
• 24:32
  Speaker 0: and so the interesting thing about the human reference that this is that it is multi-dimensional... ...there is spatiality.
• 24:39
  Speaker 0: I mentioned eighty one three D reference organs.
• 24:41
  Speaker 0: we have a tonne off two D and three D data sets had been mapped into it but then its also abstract.
• 24:48
  Speaker 0: Each of the anatomical structures, for example think off.
• 24:52
  Speaker 0: For example The left ventricle of the heart which is one of the big chambers that holds blood Is an anatomica structure and it is associated with what's called an ontology term?
• 25:02
  Speaker 0: An ontological term?
• 25:03
  Speaker 0: It just a number That denotes the left ventricular of the hard.
• 25:07
  Speaker 0: because you know some people call at the last ventricles others call in the left integral of their hearts on.
• 25:11
  Speaker 0: ontology helps us to give the same idea too Terms that many people call differently.
• 25:19
  Speaker 0: And so this connection between, you know spatial three-dimensional Multidimensional entities and kind of abstract ontology terms is very important for the human reference atlas.
• 25:31
  Speaker 0: What?
• 25:31
  Speaker 0: That means for us when we develop VR software Is that we have a whole bunch off three dimensional entities that user can see but We kind of ever hidden registry of knowledge as what ontologies are knowledge presentations about them.
• 25:44
  Speaker 0: And that is what makes this kind of similar to a virtual world or video game.
• 25:49
  Speaker 0: now, What my team and I are doing is we're developing an organ gallery for the human reference atlas That allows users to explore data sets in VR.
• 26:00
  Speaker 0: specifically you mentioned earlier The HRA powers-of-ten study.
• 26:04
  Speaker 0: This is where we banded together with six other early career scientists To take their data and show it in virtual reality harmonized to that human body we're building there.
• 26:33
  Speaker 2: Yeah,
• 26:52
  Speaker 0: it's always hard to compare yourself.
• 26:54
  Speaker 0: I do that sometimes too.
• 26:55
  Speaker 0: So this is a carrying case for the meta quest.
• 26:58
  Speaker 0: three We happen to use Meta but we have used other headsets in the past and of course one Of my team members she kind of understanding duty.
• 27:07
  Speaker 0: two scour the news For upcoming hardware releases were especially looking forward right now to a valve The company that produces steam.
• 27:16
  Speaker 0: They have announced a new headset called the Steam Frame, and of course we always make sure to keep up with the hardware that's coming.
• 27:24
  Speaker 0: We are not hardware developers!
• 27:26
  Speaker 0: And...we're not in business building our own VR headsets so- This would also
• 27:30
  Speaker 1: be very expensive to do or?
• 27:32
  Speaker 0: Potentially but it is just different kind of skill set.
• 27:35
  Speaker 0: Yeah,
• 27:35
  Speaker 2: it's a different line of work.
• 27:37
  Speaker 2: Sure Of course and
• 27:38
  Speaker 0: so we depend on kind of what's coming down the pipe in what they can get off the shelf.
• 27:43
  Speaker 0: So as I open this again This is all that i need now.
• 27:46
  Speaker 0: so I take these everywhere with me right?
• 27:48
  Speaker 0: Um, so this Is What I um.
• 27:51
  Speaker 0: whatever
• 27:51
  Speaker 1: case in case reality gets boring
• 27:53
  Speaker 0: exactly yeah.
• 27:55
  Speaker 0: And also just sometimes you know busted out somewhere to confident like hey Can I show You this for five minutes.
• 27:59
  Speaker 0: The nice thing is That people are usually quite Open To It which is a nice thing, that's one of the big advantages.
• 28:06
  Speaker 0: People are usually pretty happy to try these on.
• 28:09
  Speaker 0: so this here is the actual headset.
• 28:12
  Speaker 0: and think about it as an gaming console you can wear in your head.
• 28:16
  Speaker 0: what do we see at the front?
• 28:18
  Speaker 0: Our cameras allow our headsets essentially track itself.
• 28:22
  Speaker 0: It also allows the headset give us view from outside world.
• 28:27
  Speaker 0: So I could actually use them while doing dishes for example Which is kind of silly, but it's something that's doable.
• 28:36
  Speaker 0: People will sit on the couch and watch a movie with this because having your own gigantic TV you don't have to mount on wall always there doesn't break etc.
• 28:46
  Speaker 0: What we can also see here are quite few in our lab.
• 28:50
  Speaker 0: We give them names.
• 28:51
  Speaker 0: This one is Judy A. We named them after characters from the video game Cyberpunk, that you might have heard about.
• 28:59
  Speaker 0: The older generation meta headsets we used to have were named after the Matrix Characters.
• 29:05
  Speaker 0: So when they get their next headset... ...we'll think which franchise we're going use for naming.
• 29:10
  Speaker 0: then Then of course a charger with two AA batteries that go into the controller And this is where it gets especially interesting.
• 29:23
  Speaker 0: I often have to explain the people when i show them all VR hardware, how they actually move in there.
• 29:29
  Speaker 0: How they interact with three d objects etc.
• 29:32
  Speaker 0: and The way i explained it to them is Have you ever used a playstation or an xbox controller?
• 29:38
  Speaker 0: And if so It's just like you take one of them and rip them in half and then you have one per hand.
• 29:45
  Speaker 2: Makes sense.
• 29:46
  Speaker 2: Yeah,
• 29:47
  Speaker 0: yeah.
• 29:48
  Speaker 0: and then you know I explained to them how you move with the joysticks And it always takes a bit of onboarding that people generally get in support fast
• 29:56
  Speaker 2: now?
• 29:56
  Speaker 0: Yeah, and then You know Of course we're not reinventing The wheel here.
• 30:00
  Speaker 0: video games In the early two thousands such as halo for example have basically solidified the way That you use joysticks To Move where you Know what left joystick is used For forwards backwards sidewards movement and right hand Is Used for rotating.
• 30:15
  Speaker 0: if i remember it might be switched the other way but there's a de facto standard for this and we're just replicating that standard.
• 30:22
  Speaker 0: And then, This is not a mouse.
• 30:24
  Speaker 0: so rather than having an x and y position and two buttons you have an XYZ position also in XYZ rotation so its inherently multi-scale multidimensional input.
• 30:35
  Speaker 0: You also get two buttons that you would also get on a video game controller.
• 30:40
  Speaker 0: And then, you get the trigger button under your index finger and the grip button in the inside.
• 30:45
  Speaker 0: You can use this to mimic movements you'd make with your hands such as grabbing something... Yeah!
• 30:51
  Speaker 0: ...and it always takes people a few seconds to wrap their head around these.
• 30:56
  Speaker 0: It's especially disorienting when we're in VR and don't actually see our hands or representations of the controller.
• 31:02
  Speaker 0: But it just takes a bit of onboarding to teach this.
• 31:05
  Speaker 0: And again, in many cases we're not doing all the work again.
• 31:08
  Speaker 0: there's plenty of great VR games that have gone through all of the trouble of explaining rather complicated control schemes where you can grab stuff out and move back into virtual tool belts than use for inspiration.
• 31:26
  Speaker 1: because if want get closer reality, right?
• 31:31
  Speaker 1: I mean at a point where...
• 31:35
  Speaker 0: Yeah you don't want to move so far away from what you would do physically with your body that it becomes completely not understandable.
• 31:41
  Speaker 0: There's like a little theory that i have of input abstraction.
• 31:46
  Speaker 0: So if you are in the video game and let say you wanna open a door which actually programming doors is surprisingly complicated video games What you will do on a video game with a two-D controller is You go there you press a button to open the door and then the door opens.
• 32:02
  Speaker 0: So all of that very complicated business, extending your hand grabbing the handle moving it down along the hinge pushing opened the door in an inch and walking through is abstracted away into a buttonpress.
• 32:15
  Speaker 0: In VR You could do The Boring Thing And just also map It To A Button Press Or Could Go To The Interesting Thing Of.
• 32:22
  Speaker 0: Actually You Know Mimicking The opening of the actual door and that's one of the beautiful things where you can get much closer to the natural human input.
• 32:31
  Speaker 2: It immediately comes into my mind in use cases for science or for scientific purposes.
• 32:39
  Speaker 2: just come to mine, but I used to work with a institution that does further education for surgeons on the VR side of thing so i can well imagine what they were doing to simulate like heart disease living pigs that they operate on.
• 32:55
  Speaker 2: Of course, afterwards the pigs are dead and maybe this could be replaced.
• 33:00
  Speaker 2: if you go in a microscopic spheres of programming VR?
• 33:06
  Speaker 2: Will it possible eventually or soon-ish?
• 33:09
  Speaker 0: There already commercial platforms out there that integrate medical VR into surgery planning and surgery training.
• 33:18
  Speaker 0: one of the big challenges is when you perform surgery or do very fine-grained movements with your hand.
• 33:26
  Speaker 0: I'm a sailor, so for example if you want to simulate what it is like to sail the boat and make very minute repairs on hardware pieces... What's lacking in VR are the haptic feedback.
• 33:37
  Speaker 0: Now there's a lot that can be done now with haptic motors.
• 33:40
  Speaker 0: You actually author a complex haptic feedback mechanism but still way abstracted from touching a surgical instrument and, you know getting the actual feedback of the material that your interacting with.
• 33:55
  Speaker 0: One thing I saw at Sony when i was Tokyo headquarters there they had stylus attached to a motor.
• 34:04
  Speaker 0: You could dip it into virtual fluid And then depending on the viscosity of the fluid The mechanical arm would give you certain type of feedback.
• 34:15
  Speaker 0: But this is a very different implement from what we have here.
• 34:19
  Speaker 0: But for certain surgical tasks, even something like just the steps needed to prepare our surgery room or you know the pre and after care of a patient absolutely this is already being used.
• 34:34
  Speaker 2: I would make the distinction between the VR side things further education as i mentioned but then real time situation in the OR where I can well imagine that on the side of AR, the possibility will evolve.
• 34:50
  Speaker 2: That you have a screening off the actual organ that's being treated in real time as you operate right?
• 34:58
  Speaker 2: Yeah.
• 34:58
  Speaker 1: and i think also for using medical devices augmented reality even for home use or complex machinery just having an augmented Option that I mean not always a technician has to come to repair something.
• 35:12
  Speaker 1: Yeah, which actually also for old people and cell phones would
• 35:16
  Speaker 2: be
• 35:19
  Speaker 1: broken.
• 35:21
  Speaker 0: That's one of the beauties of AR.
• 35:23
  Speaker 0: is that it kind of gives you superpowers.
• 35:25
  Speaker 0: It allows you to see through walls See into bodies C connections that are otherwise invisible.
• 35:31
  Speaker 0: And the other beauty about AR since we talk about that now is ar runs beautifully on phones.
• 35:37
  Speaker 0: most All modern operating systems for phones have to a certain degree, depending on the browser built in support of augmented reality.
• 35:46
  Speaker 0: On web browsers which means that you can run AR software without the user having to install anything.
• 35:52
  Speaker 0: In many cases they just have to scan a QR code Which takes them to website and runs the AR software.
• 35:58
  Speaker 0: It needs access to camera.
• 36:01
  Speaker 0: That is also something where then You don't even need fancy hardware like this, most people already have smartphones.
• 36:08
  Speaker 0: And then especially the population groups that you mentioned could easily use these kinds of interface modalities.
• 36:15
  Speaker 0: but going back to your point about surgery they are all ready systems that allow you do for parts.
• 36:22
  Speaker 0: specific types very well written out surgeries follow strict procedures and there's also laparoscopic surgeries which has existed are kind of a similar in that vein.
• 36:35
  Speaker 0: There's, of course always a gap between developing a cool prototype and actually getting it into a surgical room which oftentimes also involves you know getting a surgeon on your side because people who perform surgery they need full control and everything you introduce could potentially disturb their control is going to have very low chance at being successful
• 37:10
  Speaker 2: Easy.
• 37:11
  Speaker 2: I would propose, too.
• 37:12
  Speaker 2: for the remaining time we have to do a little rapid fire round with some crucial questions that are still on the table?
• 37:20
  Speaker 1: I find it very interesting.
• 37:22
  Speaker 1: will We all wear smart glasses in ten years?
• 37:26
  Speaker 0: i don't think so.
• 37:27
  Speaker 0: Can i say why?
• 37:28
  Speaker 0: yeah Yeah So so.
• 37:30
  Speaker 0: i think there's three reasons.
• 37:31
  Speaker 0: one is The privacy aspect.
• 37:34
  Speaker 0: i dont Think It Will Ever Be Fully Okay, to potentially be recorded at all times.
• 37:39
  Speaker 0: I mean we've had smartphones for almost twenty years now and it is still not acceptable For someone just film you in public?
• 37:46
  Speaker 0: We have NOT gotten uses.
• 37:47
  Speaker 0: i don't think this gonna change with smart glasses.
• 37:50
  Speaker 0: the second reason i think Is there is not always a need To see augmented Information about your environment as soon thing.
• 37:59
  Speaker 0: that's how we are wired.
• 38:01
  Speaker 0: And the third one will still Be probably the pricing factor.
• 38:04
  Speaker 2: Is there a regulatory framework that you could think of, that will be able to tame what is touching privacy matters for instance in ARBR?
• 38:17
  Speaker 0: That's great question.
• 38:18
  Speaker 0: I'm not an expert and i'll say there are regulatory frameworks with these kinds of intrinsic motivations where they have to follow the rules.
• 38:25
  Speaker 0: otherwise there would be punishment but also more important intrinsic motivation being good citizen and adapting to the environment that you're in.
• 38:37
  Speaker 0: And I think those two are gonna have to work together.
• 38:40
  Speaker 1: Does the metaverse already exist?
• 38:42
  Speaker 0: Yes, i do think we already have metaverses!
• 38:45
  Speaker 0: There's a very concrete story here.
• 38:46
  Speaker 0: so there is Fortnite which has an extremely popular platform.
• 38:50
  Speaker 0: or roadblocks or minecraft.
• 38:52
  Speaker 0: these are basically already meta-verses.
• 38:54
  Speaker 0: like people build their own games they make connections day by day spend significant amount of life time there.
• 39:01
  Speaker 0: twenty years ago We had second live which had a very similar kind of vibe to it.
• 39:06
  Speaker 0: And for Fortnite, so in August or July twenty-twenty I was off course the US couldn't leave could visit here and my brothers and i were like what can we do together?
• 39:17
  Speaker 0: So In fortnight there was concert by the rapper Travis Scott.
• 39:23
  Speaker 0: It's essentially...I have no idea how this would be.
• 39:25
  Speaker 0: This is probably going to pretty silly But then We actually joined a server together where The Concert happened at a set time.
• 39:33
  Speaker 0: And what happened in there was, you had the whole map full of players and the avatar of Travis Scott.
• 39:41
  Speaker 0: It's a gigantic avatar like a giant just dancing across kind-of... ...the world!
• 39:47
  Speaker 0: This was the closest I ever felt to being in a metaverse.
• 39:51
  Speaker 0: There wasn't even VR headset required So this seems very far away now.
• 39:56
  Speaker 0: But at that time it makes me understand why the metaverse Could work.
• 40:01
  Speaker 0: Why it could work?
• 40:02
  Speaker 0: Maybe not with VR headsets, but why the concept of a shared space that is just websites and chat rooms But there's actually three D That you can move in or have worked.
• 40:14
  Speaker 0: because back then this was kind Of The perfect little Petri dish to observe This environment And I still remember quite fondly.
• 40:22
  Speaker 0: It Was like A one two song Kind of thing felt distinctly spatial.
• 40:29
  Speaker 0: Very
• 40:29
  Speaker 2: interesting, which also in one go answers the question because we announced it at the beginning.
• 40:35
  Speaker 2: Metaverse is not a trademark of Mark Zuckerberg.
• 40:39
  Speaker 2: The expression It's Not A Company but something else possible space if you will.
• 40:46
  Speaker 1: And maybe we can wrap up with one question, because I think that's something that has led through this episode.
• 40:52
  Speaker 1: What can scientists learn from gamers when it comes to extended and virtual reality?
• 40:57
  Speaker 0: That is also an excellent question!
• 40:59
  Speaker 0: I think gamers and scientists are not all that different... ...I think both have a innate sense of curiosity.
• 41:07
  Speaker 0: This curiosity can be either spatial- hey i wonder what's beyond the hill I wonder what's inside the data that they just spent four months creating and you know, it costs millions of dollars to make or It could be completely abstract.
• 41:20
  Speaker 0: How can i solve this very particular puzzle?
• 41:22
  Speaker 0: That i'm confronted here with?
• 41:24
  Speaker 0: And how can i make sense off this data that i've collected?
• 41:27
  Speaker 0: What kind of patterns are in there?
• 41:29
  Speaker 0: What kind?
• 41:29
  Speaker 0: if irregularities what can learn from the outliers?
• 41:32
  Speaker 0: I don't think those two groups Are all that different from each other.
• 41:35
  Speaker 0: already trusting that sense of curiosity and following it both spatially but also in an abstract kind.
• 41:46
  Speaker 2: Thank you so much for taking us on this journey which was really outstanding, And a lot folks I think adore opener to rethink or begin thinking about the topic.
• 41:59
  Speaker 2: Also in science We wish all bests that will happen soon that mindsets change brains get more open of ER, AR spaces which is really amazing especially in the scientific community because there's a lot to meet.
• 42:17
  Speaker 2: I personally was very happy that you said not everything Because I see a little danger that in the three D world everything is haptic.
• 42:34
  Speaker 2: Everything is video You don't need text anymore and stuff like that.
• 42:38
  Speaker 2: But maybe we didn't talk about those kind of ethics And i think it wasn't the right place for them, but just wanted to mention going out.
• 42:45
  Speaker 2: Maybe We have
• 42:48
  Speaker 0: Yes,
• 42:52
  Speaker 2: or we come to Indiana.
• 42:53
  Speaker 2: That would also
• 42:54
  Speaker 0: be
• 42:54
  Speaker 2: nice actually.
• 42:56
  Speaker 0: We love it too!
• 42:56
  Speaker 0: We have podcast studios in Bloomington.
• 43:00
  Speaker 2: Wonderful again thanks for being here and we really enjoyed that.
• 43:06
  Speaker 2: Please follow the liner notes.
• 43:07
  Speaker 2: If you want to get in touch with Andy, of course we'll have his space included there and further
• 43:12
  Speaker 1: link some cool applications videos all this looks Of course that
• 43:18
  Speaker 2: right yeah And further information.
• 43:20
  Speaker 2: as always on science tales calm.
• 43:24
  Speaker 2: Thank You for being with us.
• 43:25
  Speaker 2: see around next time bye.