Latest / The Adults In The AI Room / The Power of Visualization in AI: From Compliance to Finance with Bob Levy - The Earley AI Podcast with Seth Earley - Episode #048

Transcript

The Power of Visualization in AI: From Compliance to Finance with Bob Levy - The Earley AI Podcast with Seth Earley - Episode #048 · The Adults In The AI Room

• 0:00
  SPEAKER_00: This is the Early AI Podcast, hosted by Seth Early Christmas.
• 0:05
  SPEAKER_00: Join us as we delve deep into the passions, expertise, and experiences of top leaders and practitioners to talk about what's possible with artificial intelligence.
• 0:14
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science, the digital transformation journey with design and deployment of innovative technology solutions, as well as CMS wire.
• 0:24
  SPEAKER_00: Now, enjoy the show.
• 0:28
  SPEAKER_02: Well, welcome to the Early AI Podcast.
• 0:31
  SPEAKER_01: I'm Seth Early and I'm Chris Featherstone.
• 0:34
  SPEAKER_02: And we're really excited to introduce our guest today to discuss simplifying complex data visualization and understanding AI and data science and multiple dimensions around machine learning model, dimensionality reduction, and visualization of vector embeddings for things like hate speech, and we'll talk about other examples, understanding the risks of oversimplifying complexity with organizational and financial context.
• 1:01
  SPEAKER_02: Our guest today is founder and CEO of Immersion Analytics.
• 1:05
  SPEAKER_02: He also serves in the board of directors for LifeRAFT, Inc., and he has extensive experience in RD and product management at companies like IBM and Rational Software, who's also with MathWorks.
• 1:17
  SPEAKER_02: Our guest is an expert in AI and data visualization.
• 1:19
  SPEAKER_02: He's been a speaker at prestigious events such as MIT Technology Review, O'Reilly Strata Data Conference.
• 1:26
  SPEAKER_02: Bob Levy, welcome to the show.
• 1:29
  SPEAKER_03: Great to meet you.
• 1:32
  SPEAKER_03: Great to be on the show.
• 1:33
  SPEAKER_03: Happy to answer any questions.
• 1:36
  SPEAKER_02: Sure.
• 1:36
  SPEAKER_02: So why don't we start off with I wanted to kind of get a sense of, you know, what are what are what are the misconceptions around machine learning and data and artificial intelligence that you come across?
• 1:50
  SPEAKER_03: Yeah, the one, the big one we focus around is that there's a wide belief that complexity is somebody else's job.
• 2:00
  SPEAKER_03: So if you're sitting in the boardroom, you assume you've delegated grappling with complexity.
• 2:05
  SPEAKER_03: Unfortunately, the people you've delegated it to don't have the same perspective as you would in the boardroom.
• 2:12
  SPEAKER_03: Um, so oversimplification uh leads to wrong conclusions and mistaken outcomes.
• 2:18
  SPEAKER_03: The view that I can just create a dashboard and people will pivot and look at the data from different perspectives that they will actually find what they need.
• 2:30
  SPEAKER_03: So these are some of the misconceptions we grapple with.
• 2:34
  SPEAKER_02: And how does simplifying the complexity address the misconceptions around uh AI being unuxplainable in a black box, and especially with you know highly complex large language models?
• 2:48
  SPEAKER_03: Right.
• 2:49
  SPEAKER_03: So the ability now, uh, we have a technology for visualizing up to 18 dimensions at the same time.
• 2:57
  SPEAKER_03: And an example of how that can be applied to large language model governance.
• 3:02
  SPEAKER_03: I can show this uh example as well.
• 3:05
  SPEAKER_02: And for those of you just listening in today, we do have a lot of visualizations, and we're going to so you can uh take a look at the YouTube video that we will be posting.
• 3:15
  SPEAKER_03: So, what's really exciting here is you've factors from moderation APIs.
• 3:21
  SPEAKER_03: So looking at the response text of so if you think of each data point as a prompt and response pair, there are a lot of things you can measure about those prompt response pairs in a large language model.
• 3:35
  SPEAKER_03: Uh so in this in this visual, each of the data points again represents a prompt response pair.
• 3:41
  SPEAKER_03: Now, we have this technology of leveraging special effects to show different attributes of the data.
• 3:48
  SPEAKER_03: Now, Seth, are you able to see describe this?
• 3:51
  SPEAKER_02: This is a three-dimensional view of what look like spheres, and the spheres are pulsing in different ways.
• 3:59
  SPEAKER_02: Um, there's uh uh some coloring, one of them is glowing.
• 4:03
  SPEAKER_02: Um, there's some uh smaller um dots that are surrounding another um sphere that also has a red circle around it, just giving people a sense of what types of things we're we're looking at.
• 4:15
  SPEAKER_03: So, Seth, do you notice in the visual, one of the one of the data points is glowing very intensely, another has these orbiting uh particles, another is solid, whereas the remainder are translucent.
• 4:30
  SPEAKER_05: Yeah.
• 4:31
  SPEAKER_03: And what that means, so these are measuring different factors from a moderation API.
• 4:37
  SPEAKER_03: So factors including self-harm, violence, explicit content, hate speech, um, all using standard moderation APIs.
• 4:48
  SPEAKER_03: The technology can also be used to layer in sentiment analysis analysis scores, natural language processing algorithms, uh, dictionary or neural net-based uh topic categories.
• 5:01
  SPEAKER_03: You can layer in if you have anonymized user demographics uh for the purpose of making sure the model responses are level across all uh demographic groups.
• 5:15
  SPEAKER_03: Uh you can also layer in confidence levels and other quality assurance metrics.
• 5:19
  SPEAKER_03: So, what's really powerful here is you've noticed the uh the glowing point that indicates hate speech.
• 5:26
  SPEAKER_03: You've noticed the point with satellites and you've noticed the solid point.
• 5:30
  SPEAKER_03: Uh, what this, these points are organized in a special way.
• 5:34
  SPEAKER_03: So we apply a standard method called principal components analysis to reduce the vector embeddings of the response.
• 5:44
  SPEAKER_03: So this is sometimes a thousand or more dimensions.
• 5:48
  SPEAKER_03: It turns out there are standard math techniques, so principal components analysis that identifies feature importance that uses linear algebra to identify correlated components, and it synthetically reduces this part of your data set so that it can be used to organize the data points on an X, Y, and Z plane.
• 6:11
  SPEAKER_03: So spatially organizing these, you can then click on the individual points, and we're contemplating an application where by clicking the neighboring points to an anomaly, you can actually read the prompt, read the response.
• 6:28
  SPEAKER_03: And even though the moderation API didn't flag it, uh, what we've seen uh empirically is that you will you will then find um potential um paths of misalignment of the model so that you can broaden, appropriately broaden any guard railing efforts that uh that those anomalies might uh so Bob, just to ask a question.
• 6:52
  SPEAKER_01: So where the sphere is in the X, Y, and Z is its um its relevancy, and then where the other spheres are around it are their association via the vectors to that relevancy.
• 7:04
  SPEAKER_01: Is that correct?
• 7:04
  SPEAKER_03: Yeah, think of the placement on X, Y, Z as a representation of how similar the points are, yeah, based on their vector embeddings.
• 7:14
  SPEAKER_02: Yep, yeah, okay, perfect.
• 7:16
  SPEAKER_02: And then the pulsing is what uh demand, what uh uh does that represent?
• 7:21
  SPEAKER_03: What feature uh so we were looking at uh pulse frequency was the absence of hate speech.
• 7:29
  SPEAKER_03: Absence example.
• 7:31
  SPEAKER_03: So more pulsing is less hate speech, uh more pulsing is less hate speech, and that reminds me, why don't I go ahead and actually um invert that mapping?
• 7:46
  SPEAKER_03: Okay, so we have um yes, less pulsing is more uh uh uh hate speech.
• 7:57
  SPEAKER_02: And then uh again the satellites, what did those represent in this uh model?
• 8:01
  SPEAKER_03: In this case, the satellites are showing uh the intent for self-harm, so that would be a response that is uh guided someone to to harm themselves.
• 8:13
  SPEAKER_02: Wow, yeah, very interesting.
• 8:15
  SPEAKER_02: And the glow, yeah.
• 8:17
  SPEAKER_03: Yeah, uh this glowing point that you see.
• 8:20
  SPEAKER_03: So glow is uh explicit content, and then there is a neighboring point that's also glowing, and we we contemplate adding an interface where you can uh explore by clicking on the neighboring points to read the prompt and read the response.
• 8:36
  SPEAKER_03: So you can see are there subtle threads of the misalignment this anomaly represents near it, and if if so, uh the proximity can be used as a way of um informing your guardrailing efforts.
• 8:51
  SPEAKER_01: Yeah, this would be interesting against like a gaming you know context, right, where you have potential hate speech that's not necessarily bad, but it's associated with these things and associated with things like grooming techniques, right, that often occur.
• 9:07
  SPEAKER_01: So that's interesting.
• 9:10
  SPEAKER_03: Can be can be shown.
• 9:12
  SPEAKER_03: Uh we talked about principal components analysis earlier.
• 9:15
  SPEAKER_03: This technique, and it's the staple of the industry, it or t-SNE, these dimensionality reduction techniques are all about taking uh a lot of attributes and boiling them down to using feature importance and matrix or linear algebra to identify correlated components.
• 9:36
  SPEAKER_03: Uh, the problem with those techniques, and the reason they can't be applied to the moderation factors like they were to the embeddings, is because they no longer contain information about specific factors.
• 9:50
  SPEAKER_03: So, in other words, you'd see an anomaly, but you wouldn't know is it hate speech, is it explicit content, is it um violence or or otherwise?
• 10:00
  SPEAKER_02: Right.
• 10:01
  SPEAKER_02: Now, how could this could this also be used in um a compliance application where someone is looking at you know regulatory compliance uh uh regulations and then looking at policies or how would how would something like that uh work?
• 10:16
  SPEAKER_02: What would the uh application be?
• 10:18
  SPEAKER_02: Or maybe there's another there's another uh visualization for that.
• 10:22
  SPEAKER_03: Absolutely.
• 10:23
  SPEAKER_03: Uh so this visual can be uh fed with real-time streaming data.
• 10:28
  SPEAKER_03: So you could envision uh a real-time streaming view of prompts and responses coming into a system, uh, an archive data set that could be then uh replayed over time.
• 10:41
  SPEAKER_03: And the idea is that by seeing um uh a wide swath of time, you'd be able to, you'd be able to explain to a regulator in what ways is the system behaving.
• 10:56
  SPEAKER_03: Uh, did it did it, was it adapted to behave more effectively uh as the result of some corrective action uh that had been taken?
• 11:06
  SPEAKER_03: So it's a good visual tool to explain that for uh a regulator who most likely is not an expert in AI technology at the deepest level.
• 11:17
  SPEAKER_01: So let me ask you this then.
• 11:19
  SPEAKER_01: You know, beyond you know, some of these types of visualizations, how would this help in terms of like evaluating model performance?
• 11:28
  SPEAKER_03: So this can be used with so why don't we look at a machine learning example?
• 11:36
  SPEAKER_03: Uh so this comes from Kaggle, uh common data science website.
• 11:40
  SPEAKER_03: It's the stroke prediction challenge.
• 11:42
  SPEAKER_03: And we look at I'm sorry, the web prediction challenge?
• 11:45
  SPEAKER_03: Uh it's to predict stroke.
• 11:47
  SPEAKER_03: So patients who who had a um an arrhythmia or stroke event.
• 11:53
  SPEAKER_03: So in this training data set, the red points represent patients who had a stroke and the greens did not.
• 12:00
  SPEAKER_03: There are a number of factors behind this.
• 12:03
  SPEAKER_03: Uh, we're able to see um, though, as we layer in each model, so we'll layer in uh linear regression as the orbiting satellites.
• 12:15
  SPEAKER_03: And what we notice is that it identified only two of the data points here and here.
• 12:23
  SPEAKER_03: Uh, we would want to see that that kind of uh model or any model correctly identify all of the red points and not identify the green points.
• 12:33
  SPEAKER_03: Now we can layer in other model types as well.
• 12:36
  SPEAKER_03: So if we layer in random forests as opacity, we'd want to see the red points stay solid and we'd want to see the green points go translucent.
• 12:47
  SPEAKER_03: Now we see a much better fit.
• 12:50
  SPEAKER_03: Now, in analyzing models, error statistics are often used, but then again, back to the point of why summary statistics are dangerous.
• 12:59
  SPEAKER_03: Um, if you rely on summary statistics, you can't have a conversation about a specific data point where things went wrong.
• 13:08
  SPEAKER_03: And what we found is that by looking at um the individual data points with your team of experts, you can have thoughtful discussion about why did the model not get this one right?
• 13:19
  SPEAKER_03: Why did the model get these other ones right?
• 13:22
  SPEAKER_03: And you can be specific uh about those elements as opposed to just offering up summary statistics.
• 13:29
  SPEAKER_02: So isn't that a risk, no matter what you're doing, uh, of losing the granularity and losing some of the uh factors?
• 13:38
  SPEAKER_02: You know, you're using summary statistics and uh you have very complex multidimensional uh data.
• 13:44
  SPEAKER_02: And uh the question is you're you're you're always there's always uh loss of data, right?
• 13:50
  SPEAKER_02: There's always some loss.
• 13:51
  SPEAKER_02: And and so how how are you, you know, is it a matter of trying different combinations or different approaches to um the the dimensionality reduction?
• 14:01
  SPEAKER_02: Uh how how does that work?
• 14:03
  SPEAKER_03: Because we have something, you know, that's why our company exists.
• 14:06
  SPEAKER_03: So think of it as a cognitive upgrade that expands your ability for if you think of wide data and also large data.
• 14:15
  SPEAKER_03: Um so by rendering different effects around data points, up to 18 of them, uh, this can be shown.
• 14:23
  SPEAKER_03: But then what do you do if you have hundreds or thousands of dimensions, right?
• 14:29
  SPEAKER_03: So in those cases, we look for clever ways to use dimensionality reduction uh where appropriate while preserving the the dimensions that there are, like for example, in this case, looking at did the model perform yes or no?
• 14:46
  SPEAKER_03: In the last example, looking at the different moderation uh factors or or quality assurance metrics.
• 14:58
  SPEAKER_02: Interesting.
• 14:59
  SPEAKER_02: Yeah, it it really is incredible uh stuff.
• 15:03
  SPEAKER_03: One of our advisors uh he teaches at MIT, he quantified this use of effects to visualize 18 dimensions uh as multiplying your odds by 3,060 times that you will notice important relationships.
• 15:21
  SPEAKER_03: And the math behind that, if if your audience is interested, um it's the number of um dimensions shown in a standard plot on a dashboard.
• 15:32
  SPEAKER_03: So in most cases that's X, Y, size, and color, so that's four.
• 15:36
  SPEAKER_03: Uh then combinatorics, four choose 18.
• 15:41
  SPEAKER_03: So it's uh 3,060 unique combinations that that one visual uh contains.
• 15:48
  SPEAKER_01: We we talked about it before the show, but you know, like I told you I have that book that says how to lie with statistics, right?
• 15:54
  SPEAKER_01: And I think you have another visualizations that show what aggregations can do, right?
• 16:00
  SPEAKER_01: When you actually, you know, if you don't look at all the data.
• 16:03
  SPEAKER_01: I don't know if you want to you know pull that chart or graph up.
• 16:06
  SPEAKER_01: And it's it's really interesting, right, for those that are listening centered around, you know, when we look at X, Y coordinates and doing summary statistics, how you can actually see, you know, very similar statistics provide a wide range of outputs.
• 16:21
  SPEAKER_01: But anyway, go ahead, Bob.
• 16:23
  SPEAKER_03: Absolutely.
• 16:24
  SPEAKER_03: Now, this this uh visual uh credited to the research team at Autodesk, they call it the uh datasaurus.
• 16:32
  SPEAKER_03: Uh, there's another uh famous French mathematician, Anascambi, produced Anascambi's quartet, which makes a similar point.
• 16:39
  SPEAKER_03: But here we have some data and we have some summary statistics off to the right: the mean, the standard deviation, the correlation.
• 16:46
  SPEAKER_03: Now we'll animate different data through this.
• 16:48
  SPEAKER_03: And as I do this, keep an eye on the summary statistics.
• 16:54
  SPEAKER_03: You'll notice with these vastly different data sets, the summary statistics are identical.
• 17:00
  SPEAKER_03: And this is one of the most important reasons why seeing all of the data points, seeing as much of the attributes, seeing the complexity is so important.
• 17:12
  SPEAKER_02: So for those of you listening, uh, one of the representations of the data is a Tyrannosaurus.
• 17:20
  SPEAKER_02: So that's why they called it the data source.
• 17:22
  SPEAKER_02: And uh the other representations, do you want to just describe one one looks like concentric circles, one looks like a star?
• 17:29
  SPEAKER_02: And again, what is it that is causing those renderings to appear as they they are?
• 17:35
  SPEAKER_03: Uh so I don't have the algorithm the Autodesk team used to generate these.
• 17:42
  SPEAKER_03: Um, but it's it's several visuals, uh sorry, several uh data points.
• 17:52
  SPEAKER_05: Yeah, it's okay.
• 17:54
  SPEAKER_03: Yeah.
• 17:57
  SPEAKER_03: So it's a number of different arrangements, be it horizontal lines, vertical lines, a picture of a dinosaur, a picture of uh other other aspects, circles.
• 18:08
  SPEAKER_03: The data points are arranged in such a way that the mean and the standard deviation and the correlation are very similar, even as we transition from one to one to the other.
• 18:18
  SPEAKER_02: Understood.
• 18:18
  SPEAKER_02: Okay, so that's right.
• 18:19
  SPEAKER_02: So these summary statistics are the same, but the actual data sets and the uh and the visualization is very different.
• 18:28
  SPEAKER_03: Absolutely.
• 18:29
  SPEAKER_03: Yeah, another interesting point would be so why would people want to see more than a few dimensions in the first place?
• 18:37
  SPEAKER_03: And this is a really good, really powerful visual analogy we found helps.
• 18:41
  SPEAKER_03: So when you look at this image, it looks like we're we're viewing tic-tac-toe.
• 18:47
  SPEAKER_03: It looks like red jacks just won the game that we think is tic-tac-toe.
• 18:52
  SPEAKER_03: And when we add just one more dimension, we realize the game was actually three-dimensional tic-tac-toe.
• 19:00
  SPEAKER_03: The red piece that we thought led to a win, in fact, did not.
• 19:05
  SPEAKER_03: It's blue's turn, and blue is about to place uh in in the winning position.
• 19:11
  SPEAKER_03: So the exact opposite outcome is is truth, and being able to see that requires a greater fidelity of dimensions.
• 19:20
  SPEAKER_01: Yeah, this is like when you always hear those folks say, well, you know, one plus one is always equal to two.
• 19:27
  SPEAKER_01: Well, that's not the case, right?
• 19:29
  SPEAKER_01: When you change the math, and you're, you know, one times one is always one, you know, those types of scenarios, which is exactly what you're showing here, where if you add another dimension, it completely invalidates what you're looking at.
• 19:41
  SPEAKER_01: Right.
• 19:43
  SPEAKER_03: Well, Macron, oh, go ahead.
• 19:44
  SPEAKER_01: Oh no, I was just gonna ask you, you know, I would love to to get an idea too, Bob, and maybe this is something we we um you know continue to think about.
• 19:53
  SPEAKER_01: But when do you guys believe, you and your colleagues believe that that you know, a lot of your work right now is done in the context of showing the visualizations of the data, how this is representing from the metadata of the perspective of the LMs and things.
• 20:08
  SPEAKER_01: When do you believe the LLMs will be able to produce these types of graphs and charts?
• 20:13
  SPEAKER_03: So we do have a we do have a patent on the technology for visualizing the multiple dimensions.
• 20:21
  SPEAKER_03: Uh, we are in discussion with a number of leading uh providers as to how they might uh use this technology to help their customers, for example, in fine-tuning efforts, in guardrail and governance efforts as well.
• 20:36
  SPEAKER_03: I can't speak to specific timeframes of when the technology will be integrated in some of those.
• 20:42
  SPEAKER_03: It is architected as a component technology, it can be used in a standard uh web application.
• 20:49
  SPEAKER_03: It can even be used in virtual and augmented reality devices.
• 20:55
  SPEAKER_03: Now, those are completely optional, but there's really strong research showing that by using devices like that, you actually triple the number of data points you can absorb versus on a flat screen.
• 21:09
  SPEAKER_02: Interesting.
• 21:11
  SPEAKER_02: Um so so this is really providing lots of different insights that you know traditional metrics and visualizations cannot provide.
• 21:21
  SPEAKER_02: Um, what does it take to leverage this?
• 21:26
  SPEAKER_02: And and again, what's the level of complexity for being able to uh build some of these visualizations?
• 21:33
  SPEAKER_03: You know, I should show um one of our uh examples comes from the financial industry.
• 21:40
  SPEAKER_03: So this one imagines you're a portfolio manager.
• 21:44
  SPEAKER_03: So this could be integrated with, say, your Bloomberg terminal or other environment.
• 21:49
  SPEAKER_03: So let's take someone who does not have a background in AI, does not have a quantitative research background, they're simply a fundamental investor like Warren Bowen.
• 22:00
  SPEAKER_03: It they're looking for value plays, bargains.
• 22:04
  SPEAKER_03: So in this case, we've the price earnings shown on the x-axis.
• 22:08
  SPEAKER_03: The investable universe here is the SP 500.
• 22:12
  SPEAKER_03: Uh, we've price book as color, so blue.
• 22:15
  SPEAKER_03: Uh there are blue bargains to the left is a good way to think of it.
• 22:18
  SPEAKER_03: But by layering, and what we found is by layering them one at a time using a technique we call stepwise storytelling.
• 22:26
  SPEAKER_03: So I'll layer in price sales, and our bergen-minded portfolio manager thinks they want to be toward the front of this chart and the left.
• 22:37
  SPEAKER_03: So they make a selection.
• 22:39
  SPEAKER_03: And they, as we layer in more attributes, so we can layer in book value as this uh shimmering effect.
• 22:48
  SPEAKER_03: So the more intensely it's shimmering, the greater the book value.
• 22:52
  SPEAKER_03: So Goldman Sachs and Google, uh glowing, uh, in this case to show EBITA, so Apple, uh Verizon ATT.
• 23:02
  SPEAKER_03: But then when we layer earnings per share, and we didn't know earnings per share was relevant before the analysis, but by taking a look, what we see is that that AA point that our portfolio manager would have selected, it actually has a terrible earnings per share.
• 23:21
  SPEAKER_03: And just like in that tic-tac-toe analogy, uh the point they wanted in this case was was near it.
• 23:29
  SPEAKER_03: It was a very different conclusion once they see the whole picture.
• 23:34
  SPEAKER_02: So when you start thinking of large language models that have um hundreds of thousands of dimensions, right?
• 23:42
  SPEAKER_02: I mean, what what is that those orders of magnitude of greater dimensionality mean?
• 23:48
  SPEAKER_02: Because that's what that's what the large language model is about.
• 23:52
  SPEAKER_03: Yeah, when you look inside of a large language model, you find a neural network, uh, you find a lot of linear algebra, you find individual dimensions that are generated by the system that may or may not have human meaning.
• 24:08
  SPEAKER_03: So when you're taking, like, say the vector embedding space is a really good example of this, you've thousands of dimensions.
• 24:16
  SPEAKER_03: In those cases, we found techniques like principal components is really useful as a way of absorbing those thousands of dimensions into a way of visually organizing.
• 24:28
  SPEAKER_03: But where those techniques fall short is if you then want to make specific observations, like is it hate speech, or is it um a demographic element or so forth?
• 24:40
  SPEAKER_03: So the combination of both of those turns out to be a really powerful way to uh to address um um even uh the world's widest data sets.
• 24:52
  SPEAKER_01: So, yeah, this is interesting, Bob.
• 24:53
  SPEAKER_01: I appreciate you doing this.
• 24:54
  SPEAKER_01: I mean, it's it's it's hard not to have a podcast without visualizations if we're gonna talk about visualizations, right?
• 25:04
  SPEAKER_01: And dive into it.
• 25:08
  SPEAKER_03: Um we found so explaining things like what our MIT professor advisor um uh had helped quantify, explaining concepts around virtual and augmented reality and how those help triple capacity for data points, but there's nothing quite like seeing it.
• 25:26
  SPEAKER_01: So, Bob, one of the things you brought up was the relevancy of seeing this, finally seeing this and then bringing a lot of this data to life to answer more insightful questions, questions, which you know, you talked about the augmented reality and the virtual reality type of world.
• 25:41
  SPEAKER_01: So it seems to me, and I'd love to get your take on it, that these all these goggles and glasses and all these types of scenarios will now finally have some relevance in business to helping us instead of you know, now now, hey, everybody put your goggles on, let's look at the uh the 10K, right?
• 25:59
  SPEAKER_01: You know, from an executive perspective.
• 26:01
  SPEAKER_01: So go ahead and I'd love to get your thoughts there.
• 26:03
  SPEAKER_03: Of course.
• 26:04
  SPEAKER_03: And we're seeing this already in education.
• 26:07
  SPEAKER_03: So professors at leading institutions demonstrating complex concepts to their students.
• 26:13
  SPEAKER_03: Uh, we've won using the Microsoft HoloLens.
• 26:16
  SPEAKER_03: We are a partner with Microsoft or on their it's a spatial computing device where you see the room around you and you also see the content as though it were holographic.
• 26:25
  SPEAKER_03: And everything you just saw here on the flat screen, that's represented holographically in space.
• 26:31
  SPEAKER_03: You fill your room with the data, you walk around it, and you're able to absorb much more.
• 26:37
  SPEAKER_03: Uh, one of the papers in this uh from 1994, the researcher was Colin Ware.
• 26:43
  SPEAKER_03: Uh, he was able to prove empirically that uh people were able to absorb uh three times as many data points in a pathfinding experiment around graph data than they were on a flat screen.
• 26:55
  SPEAKER_03: Now that was with VR as it existed back in '94.
• 26:59
  SPEAKER_03: It's come a long way since then.
• 27:01
  SPEAKER_03: We were um recently included.
• 27:04
  SPEAKER_03: Apple um hosted before they launched the Vision Pro, finally allowed to talk about this.
• 27:11
  SPEAKER_03: They hosted pre-release porting labs in Cupertino.
• 27:14
  SPEAKER_03: And I will say I've never seen a device with the level of visual fidelity that this one has.
• 27:20
  SPEAKER_03: If you have a chance, you know, check it out at your local Apple store.
• 27:24
  SPEAKER_03: They have uh uh fairly widely available uh demos.
• 27:28
  SPEAKER_03: And it turns out that by seeing these things with such high fidelity, you're you're missing less.
• 27:34
  SPEAKER_03: Now, another partner of ours is Zpace.
• 27:38
  SPEAKER_03: So these folks uh make the world's first 3D laptop.
• 27:42
  SPEAKER_03: So it's a standard-looking laptop.
• 27:45
  SPEAKER_03: You uh position your view right in front of the screen.
• 27:50
  SPEAKER_03: There's special cameras that track your pupils, and the screen is comprised of prisms.
• 27:55
  SPEAKER_03: So one image to one eye, a slightly different image to the other eye, and it results in an experience where it feels like the content literally is hovering over the keyboard of the thing.
• 28:07
  SPEAKER_03: Uh, I have one of them sitting behind me.
• 28:09
  SPEAKER_03: It is one of the more interesting devices out there.
• 28:15
  SPEAKER_02: But then you need you need something to uh visualize in three dimensions to uh yes to make use of it, right?
• 28:22
  SPEAKER_03: And we have that in spades, yes.
• 28:25
  SPEAKER_03: Yeah, yeah.
• 28:26
  SPEAKER_01: Well, where does where does compliance then start to play a role, right?
• 28:30
  SPEAKER_01: Because part of this is, you know, like I said, it's one thing to be able to pull a lot of this data in, you know, and make it you know simplistic or more simplistic, right?
• 28:39
  SPEAKER_01: I think that's one of the goals here is yeah, we're dealing with really complex information and situations, but essentially all you're doing is making it more um more simple, more consumable, right, and yet more understandable, right?
• 28:55
  SPEAKER_01: So is that fair?
• 28:57
  SPEAKER_03: I think it's perfect.
• 28:58
  SPEAKER_03: The uh you know, the view that if you can understand more, it's going to be a lot easier to govern those more complex topics.
• 29:08
  SPEAKER_03: When you can bring a regulator into the room and explain to them why, you know, a specific thing had hate speech, you made it change to the model, and the hate speech is gone.
• 29:17
  SPEAKER_03: And you can show them that before, after, and and the remediation steps that were taken.
• 29:24
  SPEAKER_03: Um the the laws, the regulations around AI are of course still emerging, yeah.
• 29:31
  SPEAKER_03: But it comes back to common sense.
• 29:34
  SPEAKER_03: These regulators need to be able to understand things that they weren't trained to understand.
• 29:40
  SPEAKER_01: Yeah, I think I read um I read somewhere that there was close to 600 different compliance standards from 89 countries or 86 countries, and it's all over the board.
• 29:50
  SPEAKER_01: There's no global standard.
• 29:52
  SPEAKER_01: And I feel like, and I'd love to get your take here, where does this type of work then does it does it take existing standards and compliances, you know, in terms into consideration, and or does it beg for difference, you know, different ones and more um, let's say more thorough introspection applies, right?
• 30:14
  SPEAKER_03: Uh so our focus as a company is around um the visualization technology that makes it easier to understand these things.
• 30:22
  SPEAKER_03: Um from a from a regulatory perspective, though, yes, these standards have to emerge.
• 30:30
  SPEAKER_03: They'll emerge as most regulations do from case law, so trial and error, essentially, which is very unnerving.
• 30:39
  SPEAKER_03: Um, I feel a lot better about a future where my 10-year-old daughter grows up into a world where she can actually understand what's going on in these systems as opposed to hoping a regulator had the right checklist.
• 30:53
  SPEAKER_03: Um yeah.
• 30:55
  SPEAKER_02: So, Bob, maybe we could go through some of the other visualizations that you have up here to think more about what are the other corporate applications that we might have.
• 31:05
  SPEAKER_02: Um, do you want to walk through some of the other ones?
• 31:07
  SPEAKER_03: Uh, so we have a number in uh finance.
• 31:10
  SPEAKER_03: So this next one comes from the derivatives space.
• 31:14
  SPEAKER_03: So think stock options.
• 31:16
  SPEAKER_03: Uh there's a a standard view in the investment management world called a volatility smile.
• 31:23
  SPEAKER_03: So we're looking at puts and calls.
• 31:25
  SPEAKER_03: The reds are the puts and the greens are the calls.
• 31:29
  SPEAKER_03: These are contracts for the ability to buy a stock at a certain price by a certain time.
• 31:35
  SPEAKER_03: So the strike price is here on the x-axis, the time to maturity or how long the options contract uh will last is is on the z-axis.
• 31:45
  SPEAKER_03: And a metric that's uh uh essentially shows the implied volatility, the volatility that the given contract would imply.
• 31:56
  SPEAKER_03: So we see this standard kind of smile uh when uh the price of the underlying stock is close to um uh is close to um uh the price of the contract, the strike price of the contract.
• 32:13
  SPEAKER_03: That's where volatility will be the lowest.
• 32:15
  SPEAKER_03: We go ahead and layer different derivative statistics.
• 32:19
  SPEAKER_03: So this one being shown a shimmer, it's referred to as delta.
• 32:22
  SPEAKER_03: So this is how much will the price of a contract change given a unit change of the price of the underlying.
• 32:30
  SPEAKER_03: So we might be looking at Microsoft or Apple or others.
• 32:33
  SPEAKER_03: How much will the price of each of these contracts change?
• 32:37
  SPEAKER_03: Now, as we look at these, we're looking for things that are out of the ordinary.
• 32:42
  SPEAKER_03: And so far we don't see it.
• 32:44
  SPEAKER_03: Uh, but we'll layer in another one.
• 32:46
  SPEAKER_03: Uh so these are called the Greeks.
• 32:48
  SPEAKER_03: Uh, theta is used as a way of showing how sensitive the price is to the passage of time.
• 32:56
  SPEAKER_03: People will pay more for a contract they have a lot more time to exercise than a contract that they must exercise immediately.
• 33:04
  SPEAKER_03: Uh, so this glowing that you're seeing uh represents theta.
• 33:10
  SPEAKER_03: And options math would tell us we expect to see high theta close to strike in contracts that are about to expire.
• 33:18
  SPEAKER_03: What we notice is that theta fades out, and then there's a hot spot of a couple of contracts in the most distant months that and this lasted for 20 minutes that are showing high theta.
• 33:31
  SPEAKER_03: This never should happen.
• 33:33
  SPEAKER_03: This is completely an arbitrage opportunity.
• 33:37
  SPEAKER_03: Interesting.
• 33:37
  SPEAKER_03: Many examples like this.
• 33:41
  SPEAKER_02: So so again, how does that what's the arbitrage opportunity in that uh representation?
• 33:47
  SPEAKER_03: Ah, yes.
• 33:47
  SPEAKER_03: So the the anomalous contracts are mispriced.
• 33:50
  SPEAKER_03: Yep.
• 33:51
  SPEAKER_03: So buying or selling uh to profit from the mispricing until the market comes back into alignment.
• 34:00
  SPEAKER_03: And again, the theta represents the it's the each contract sensitivity to the passage of an additional day.
• 34:08
  SPEAKER_03: So how much will the price decrease uh each each day it is in motion?
• 34:15
  SPEAKER_02: And it makes more sense if it's about to uh mature versus it's a long time out.
• 34:21
  SPEAKER_02: Yeah.
• 34:22
  SPEAKER_03: We'd expect to see the highest data at maturity.
• 34:26
  SPEAKER_03: We'd expect to see the lowest data distant from maturity.
• 34:30
  SPEAKER_03: Anomalies show up in ways you'd never expect.
• 34:33
  SPEAKER_03: Um personal experience.
• 34:37
  SPEAKER_03: Personal experience by seeing many dimensions at once.
• 34:41
  SPEAKER_03: It doesn't matter if it's glowing or shimmering or what it's doing.
• 34:44
  SPEAKER_03: It matters that it's doing something that you're able to see.
• 34:47
  SPEAKER_03: And that's uh that's what we have here.
• 34:50
  SPEAKER_03: That's great.
• 34:52
  SPEAKER_02: Yeah, can't carry on.
• 34:53
  SPEAKER_02: Um carry on if you want to show some other or explain some of the other because this gives us a sense of what those specific applications are.
• 35:00
  SPEAKER_01: Well, I'm just I'm just glad the discussion is is truly multidimensional.
• 35:04
  SPEAKER_01: I mean, no, no pun intended.
• 35:06
  SPEAKER_02: No, a pun was intended.
• 35:11
  SPEAKER_01: Yeah, what else do you have, Bob?
• 35:12
  SPEAKER_01: It's great.
• 35:13
  SPEAKER_02: Great, okay.
• 35:14
  SPEAKER_03: Uh so this one comes from another financial uh group.
• 35:18
  SPEAKER_03: So we were looking at uh price.
• 35:22
  SPEAKER_03: So height is showing us returns, a proxy for price, looking at different companies in the technology space.
• 35:30
  SPEAKER_03: Now there was some interest in understanding, well, why did some perform better than than others?
• 35:36
  SPEAKER_03: So if we if we look at debt capitalization as glowing, we see that encumbrance by debt is a contributor to low performance.
• 35:47
  SPEAKER_03: These are the points that did not appreciate, or the series that did not appreciate as much.
• 35:53
  SPEAKER_03: Uh, similarly, uh price sales ratio.
• 35:58
  SPEAKER_03: So we find that one of the stocks is bright red, that's Adobe, uh, and they have a uh substantially higher price sales ratio.
• 36:08
  SPEAKER_03: And that resulted in that seems to be a contributing factor or at least a correlating factor to increase performance.
• 36:17
  SPEAKER_03: So these are clues.
• 36:19
  SPEAKER_03: Um, I believe it was uh said uh correlation is not causation, but sometimes it's an interesting hint.
• 36:26
  SPEAKER_03: And this is a way of seeing those hints woven together.
• 36:31
  SPEAKER_02: That's fascinating.
• 36:33
  SPEAKER_02: And what are the smaller um uh what are the smaller pieces there that are um kind of chained together?
• 36:42
  SPEAKER_02: Yes, yes.
• 36:43
  SPEAKER_03: So we're showing a financial time series of prices each day, but um companies report um financial like fundamental metrics once a quarter.
• 36:54
  SPEAKER_03: The bigger points are showing each quarterly report, uh, and the the green points are showing us the the daily prices and this example.
• 37:03
  SPEAKER_03: Very good.
• 37:04
  SPEAKER_03: Very good.
• 37:05
  SPEAKER_02: Next one.
• 37:07
  SPEAKER_03: Great.
• 37:08
  SPEAKER_03: Um, we have had now this one works a lot better in virtual reality than it does on a flat screen, but what you're seeing here is an artificial neural network.
• 37:19
  SPEAKER_03: So you're seeing both weights and biases.
• 37:22
  SPEAKER_03: Um, when you fill the room with data, you have it physically with you, you're able to visualize interestingly large neural networks.
• 37:32
  SPEAKER_03: And you can see, for example, through a learning algorithm like backpropagation, you know, at each training epoch when you're training one of these, how do the weights and biases change?
• 37:42
  SPEAKER_03: Wow.
• 37:43
  SPEAKER_03: People have never been able to see that before.
• 37:45
  SPEAKER_03: Wow.
• 37:46
  SPEAKER_03: We're at the forefront of helping people understand the internals of these.
• 37:52
  SPEAKER_02: Then walking walking through the data will have a whole different uh meaning.
• 37:57
  SPEAKER_02: Well, like you say, if you fill the room with data and you're using virtual reality and you can walk around it and see it from different angles, that will be really something.
• 38:05
  SPEAKER_02: And you can watch the training as it's happening.
• 38:08
  SPEAKER_01: The fact that we can look at this, because we always talk about, you know, in a two-dimensional way, looking at a graph neural net for things like I deal with telco a lot, I deal with customer experience a lot, deal with those scenarios.
• 38:20
  SPEAKER_01: And you're always trying to derive out the associations, of course, and then the propensities, right, to predict what that looks like for specific things.
• 38:30
  SPEAKER_01: And you know, it'd be really interesting to get in and just look at it from a true multidimensional perspective as opposed to just guessing what that relationship looks like.
• 38:40
  SPEAKER_01: Again, man, you know, the the reason for everybody to get these um these these glasses and goggles to actually look at and actually derive value from the data.
• 38:50
  SPEAKER_03: You know, the pandemic forced us before the pandemic, we were primarily focused on virtual and augmented reality because it is quite interesting.
• 38:58
  SPEAKER_03: Um, the pandemic forced us to we we have a component.
• 39:02
  SPEAKER_03: If you have a web application, you can embed these kinds of visuals in your web application as a component technology.
• 39:10
  SPEAKER_03: Uh, so for example, if you have a platform where people build AI or govern AI systems, uh, these visuals can be made a component of those platforms fairly easily.
• 39:24
  SPEAKER_03: Um we found that the wide data problem, so seeing the special effects, works great on a flat screen.
• 39:32
  SPEAKER_03: The large data problem, that's where the glasses really help, or the 3D laptop.
• 39:38
  SPEAKER_03: Yeah, and I'm I'm sometimes asked, Do I do I see a future where you'll walk into a boardroom and people will actually have the goggles on?
• 39:46
  SPEAKER_03: And I'm very happy to see devices like the Z Space laptop, the 3D laptop.
• 39:52
  SPEAKER_03: It does not look strange.
• 39:54
  SPEAKER_03: It's the sort of thing you'd expect to see in a boardroom.
• 39:58
  SPEAKER_02: And just for people who are listening, you have to watch the uh the uh YouTube video on this because this is really uh amazing stuff.
• 40:06
  SPEAKER_02: Uh, do you want to explain the uh we have a few more minutes, the uh turban looking one or the one that looks like a uh knowledge graph?
• 40:15
  SPEAKER_03: Ah, yes.
• 40:16
  SPEAKER_03: So we'll take the graph.
• 40:18
  SPEAKER_03: This was graph data.
• 40:19
  SPEAKER_03: So again, we have this concept of a multidimensional data point.
• 40:23
  SPEAKER_03: We also have multidimensional edges where different special effects are used.
• 40:28
  SPEAKER_03: This is an analysis.
• 40:29
  SPEAKER_03: We had a Python script comb Reddit looking for different programming themes, which languages were related to which other languages in this case.
• 40:39
  SPEAKER_03: Uh, it's an example of how knowledge graphs can be represented, but anything we have numeric measures, so different attributes, we can show many of those uh per point as as well.
• 40:51
  SPEAKER_03: Now, this one, and bear with me while I restart this.
• 40:55
  SPEAKER_03: So I have a Python script running in the background that is calling Coinbase's API.
• 41:02
  SPEAKER_03: We're looking at the real-time price of different cryptocurrencies, which it's in a very, very strong um bull trend of of late.
• 41:14
  SPEAKER_03: Um, what we're looking at here is each second.
• 41:17
  SPEAKER_03: So the relative change in Bitcoin versus Ethereum versus some of these other other coins.
• 41:24
  SPEAKER_03: The color is showing us uh volume.
• 41:27
  SPEAKER_03: So we know spike in volume before price increased on um on Bitcoin, for example.
• 41:35
  SPEAKER_03: We just had a spike in volume on AVAX.
• 41:38
  SPEAKER_03: Uh that spike corresponded to an increase in its price.
• 41:43
  SPEAKER_03: So you have this one visual that's giving you the same information of it would be 28 stock charts or crypto charts on a dashboard.
• 41:53
  SPEAKER_03: Um, if you think of the the setup where you see lots of uh screens in front of these traders, they need a a way to uh um to manage the cognitive overload.
• 42:07
  SPEAKER_03: And right is one of those.
• 42:09
  SPEAKER_02: Yeah, this is uh this is perfect, it's beautiful.
• 42:11
  SPEAKER_02: So this is a second by second real-time uh pricing.
• 42:16
  SPEAKER_03: That's right.
• 42:16
  SPEAKER_03: I have a Python script running in the background that's connected to Coinbase's uh real-time uh data API.
• 42:24
  SPEAKER_03: It is um then the Python script is feeding data into our visualization engine here, and the technology is available to visualize anything you you wish.
• 42:37
  SPEAKER_03: You can use simple CSV data, you can use Python or other programming languages, you can introduce it to enterprise level applications and have the applications control uh the data flow as well.
• 42:51
  SPEAKER_01: What what does the um the overhead look like in terms of compute needs for this, right?
• 42:56
  SPEAKER_01: What GPU type?
• 42:57
  SPEAKER_03: So the visualizations.
• 42:59
  SPEAKER_03: So this piece is a little bit of trade secret, but um so my my CTO has a background in rendering technology.
• 43:08
  SPEAKER_03: So think the graphic processing unit.
• 43:11
  SPEAKER_03: Uh, we have benchmarked everything you're seeing today.
• 43:14
  SPEAKER_03: I can show you on a circa 2014 Microsoft Surface Pro 4, which is not very powerful.
• 43:23
  SPEAKER_03: Yeah, it's all all rendering is performed client-side.
• 43:28
  SPEAKER_03: And you know, with with um interesting level devices.
• 43:34
  SPEAKER_03: So I one of the machines here on loan from Intel is a server core processor with a high end graphics card.
• 43:41
  SPEAKER_03: We've visualized hundreds of thousands of data points uh with that one.
• 43:47
  SPEAKER_03: It gets down to how much data, how many data points do you really want to visualize?
• 43:53
  SPEAKER_03: Uh the bottleneck there is the visual cortex.
• 43:56
  SPEAKER_03: How much can we actually see?
• 43:59
  SPEAKER_05: Yeah.
• 44:00
  SPEAKER_03: So that's in the thousands to tens of thousands.
• 44:03
  SPEAKER_03: And at that scale, um, you can even try.
• 44:06
  SPEAKER_03: We do have a sample iPad app with some of these demos if you type immersion analytics into the app store.
• 44:13
  SPEAKER_03: And you can see you can see some of these running on a standard iPad.
• 44:17
  SPEAKER_03: And the touch interface is it's not as good as virtual reality, uh, but it's better than a regular flat screen where you have no sense of control.
• 44:29
  SPEAKER_03: People do get a broader sense of the data, even with a standard iPad.
• 44:35
  SPEAKER_02: So uh this has been fantastic, Bob.
• 44:37
  SPEAKER_02: We've really, really enjoyed this.
• 44:39
  SPEAKER_02: I know we've had other conversations and this really went into a lot of depth.
• 44:43
  SPEAKER_02: Um, so I appreciate your time.
• 44:45
  SPEAKER_02: And uh by the way, just uh so people can know a little bit more about you.
• 44:49
  SPEAKER_02: Um uh what do you do for fun?
• 44:51
  SPEAKER_02: Um what what what do you do in your spare time?
• 44:54
  SPEAKER_02: I know you have kids and any any hobbies, any uh uh leisure activities that you're uh interested in.
• 45:01
  SPEAKER_03: Absolutely.
• 45:02
  SPEAKER_03: Um, so I am an avid technologist, I like tinkering with things, fixing things, but I also love sailing.
• 45:10
  SPEAKER_03: Uh I have been known to golf as well, so uh and and travel.
• 45:16
  SPEAKER_03: Uh taking my daughter to DC um next week, actually, uh just for the first time.
• 45:22
  SPEAKER_01: So you you you pay for frustration on the golf game, that's good.
• 45:26
  SPEAKER_01: And uh if hey, let me ask you one one quick thing.
• 45:29
  SPEAKER_01: If what was what would be or what is something that uh we didn't ask you that we should have.
• 45:35
  SPEAKER_03: Okay.
• 45:36
  SPEAKER_01: Um I think we can leave with that because I think there's you know, there's this is amazing.
• 45:41
  SPEAKER_01: We can keep going, but I always like to ask that.
• 45:45
  SPEAKER_03: Oh, where to begin?
• 45:48
  SPEAKER_01: And if it's too if it's too long, no problem, right?
• 45:51
  SPEAKER_03: So we talked about the AI areas.
• 45:54
  SPEAKER_03: Uh we've talked about, I suppose, how um something like this gets integrated into standard off-the-shelf applications.
• 46:02
  SPEAKER_03: So for the technologists in the crowd who are comfortable with say uh visualization, you might think of learning GPU-based technologies and writing potentially close to a million lines of code to make some of this work.
• 46:18
  SPEAKER_03: Um, or uh we we provide standard off-the-shelf SDKs where, with in some cases, a couple dozen lines of JavaScript, you have one of these embedded in a regular uh website.
• 46:31
  SPEAKER_03: So it turns out to be a powerful way whenever an ISV has conversations or customers who have complex data that they need to understand and haven't been.
• 46:42
  SPEAKER_03: So in areas like cybersecurity or defense and intelligence, areas, of course, finance, as we've discussed, uh education, um, AI governance, of course, uh many areas we are, I think, still just scratching the surface of the applications where it fits.
• 47:01
  SPEAKER_03: That's awesome.
• 47:02
  SPEAKER_03: So, Bob, uh uh uh, where can people find you?
• 47:07
  SPEAKER_03: Immersionanalytics.com.
• 47:09
  SPEAKER_03: That's imanalytics.com.
• 47:14
  SPEAKER_03: Um, if you want to reach out by email, bob at immersionanalytics.com will reach me directly.
• 47:21
  SPEAKER_03: I also um reply to responses on our contact form on the on the site.
• 47:27
  SPEAKER_03: The site has a lot of videos that uh for those who weren't able to see uh some of the visualizations today, many videos throughout the website.
• 47:37
  SPEAKER_02: Well, thank you, uh Bob, again for your time.
• 47:40
  SPEAKER_02: This has been a pleasure.
• 47:41
  SPEAKER_02: Thank you to our audience uh for listening.
• 47:44
  SPEAKER_02: If you enjoyed this, please share the episode, subscribe, like, uh, all of those things.
• 47:50
  SPEAKER_02: And um, this has been another uh episode of the Early AI podcast, and we will see you next time.
• 48:00
  SPEAKER_01: Thanks, Bob.
• 48:01
  SPEAKER_01: Appreciate it.
• 48:02
  SPEAKER_00: Excellent.
• 48:02
  SPEAKER_00: Thank you, guys.
• 48:03
  SPEAKER_01: That's great.
• 48:06
  SPEAKER_00: Thank you for joining us in another deep dive into AI innovation.
• 48:10
  SPEAKER_00: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
• 48:16
  SPEAKER_00: The Early AI podcast is sponsored by Early Information Science and CMS Wire.
• 48:21
  SPEAKER_00: To learn more, visit early.com.
• 48:24
  SPEAKER_00: That is E A R L E Y dot com.
• 48:28
  SPEAKER_00: Thanks for tuning in.