Latest / The Adults In The AI Room / Data Tells the Story - Michelle Zhou - Earley AI Podcast with Seth Earley & Chris Featherstone - Episode # 025

Transcript

Data Tells the Story - Michelle Zhou - Earley AI Podcast with Seth Earley & Chris Featherstone - Episode # 025 · The Adults In The AI Room

• 0:03
  SPEAKER_05: Welcome.
• 0:04
  SPEAKER_05: You're listening to the Early AI Podcast.
• 0:07
  SPEAKER_05: Here we talk with thought leaders and practitioners on what's possible and practical with AI in the enterprise as we move from the current state to a future of cognitive assistance and embedded AI in all aspects of our personal and discussional lives.
• 0:22
  SPEAKER_05: Thanks so much for listening.
• 0:23
  SPEAKER_05: If this is your first time, we invite you to subscribe wherever you listen to podcasts.
• 0:28
  SPEAKER_05: And now here are your hosts, Seth Early and Chris Featherstone.
• 0:36
  SPEAKER_01: Welcome to today's podcast.
• 0:38
  SPEAKER_01: I'm Seth Early and I'm Chris Featherstone.
• 0:41
  SPEAKER_01: Chris, how is the weather there?
• 0:43
  SPEAKER_01: Is it uh getting very wintry?
• 0:45
  SPEAKER_00: Yeah, you know what?
• 0:46
  SPEAKER_00: We've got about a foot of snow on the ground.
• 0:49
  SPEAKER_00: Yeah, and we're in the, you know, in the it hovers between the um, you know, the high 20s and the teens.
• 0:58
  SPEAKER_00: So but it's a but it's dry, right?
• 1:00
  SPEAKER_00: Like they said dry.
• 1:02
  SPEAKER_00: It's not that cold.
• 1:06
  SPEAKER_00: I'm in the north, uh northwest part of Montana.
• 1:12
  SPEAKER_01: Yeah, yeah.
• 1:13
  SPEAKER_01: That's great.
• 1:14
  SPEAKER_01: That's great.
• 1:15
  SPEAKER_01: So great winter winter sports ahead for you.
• 1:18
  SPEAKER_01: Well, let's go ahead and jump into our podcast today.
• 1:21
  SPEAKER_01: Um, our next guest is passionate about creating human machine symbiosis.
• 1:26
  SPEAKER_01: She envisions a world where everyone will have his or her own AI companion, a companion who can truly understand us as unique individuals and help us in our personal and professional life.
• 1:37
  SPEAKER_01: To that end, she built a company that powers chatbots using human-centered AI.
• 1:42
  SPEAKER_01: Please welcome co-founder and CEO of Juji, Michelle Zhao.
• 1:47
  SPEAKER_04: Thank you, Sass, for having me.
• 1:49
  SPEAKER_04: And thank you, Chris.
• 1:50
  SPEAKER_04: Um appreciate the time.
• 1:54
  SPEAKER_01: So, um, Michelle, uh, give us a sense of of your background.
• 1:58
  SPEAKER_01: How uh how did you get into the space and uh tell us a little bit about that uh path?
• 2:04
  SPEAKER_04: I'm glad you asked about that, Sass.
• 2:07
  SPEAKER_04: Actually, I would say I stumbled into the computer science by accident because I both my both of my parents are physics were physicians.
• 2:19
  SPEAKER_04: So um I didn't want to become a physician, but I really want to become a biologist.
• 2:25
  SPEAKER_01: That was the time near and dear to my heart.
• 2:28
  SPEAKER_01: That's what I started out in as my undergrad.
• 2:30
  SPEAKER_04: I want to really to change them, change them humankind, right?
• 2:34
  SPEAKER_04: Advance humanity by making men pregnant so they can share the pain, the responsibilities of human reproduction with us women.
• 2:45
  SPEAKER_04: So um, however, I wasn't accepted by uh by the biology department.
• 2:51
  SPEAKER_04: Uh so um they said you had to pick the second major, otherwise you may not be accepted by the university.
• 2:58
  SPEAKER_04: So I say, hey, which one would be?
• 3:00
  SPEAKER_04: So it's almost like a through the dark the point because uh I have I had never seen a computer touch the computer before, before I decided to go to computer science.
• 3:12
  SPEAKER_04: I wasn't really fond of computer science at the very beginning because uh, you know, in computer science you have to learn the binary code, right?
• 3:20
  SPEAKER_04: Only computers understand.
• 3:22
  SPEAKER_04: You have to understand the programming language.
• 3:25
  SPEAKER_04: I remember at the beginning, many people struggled to understand why x equals to x plus one, right?
• 3:34
  SPEAKER_04: So uh now, of course, uh you after you become a programmer, you know why.
• 3:38
  SPEAKER_04: And then I really wasn't really uh fond of computer science until I came to the US.
• 3:46
  SPEAKER_04: I attended Michigan State University and to study towards my master's degree.
• 3:51
  SPEAKER_04: So there were two professors that really have changed, helped, I guess, uh, change my perspectives about computer science.
• 3:58
  SPEAKER_04: So one professor is Professor Nainang Mi.
• 4:02
  SPEAKER_04: So he gave me an opportunity to work on building a graphical user interfaces for power management, for network management, power management.
• 4:11
  SPEAKER_04: That really gave me a sense of gratification because you build a graphical user interface, people can use it, and to manage something could potentially have a real-world impact.
• 4:23
  SPEAKER_04: Then the second professor, who actually uh uh is a late professor Carl Page, and uh he really taught our AI class.
• 4:33
  SPEAKER_04: He was very passionate, right?
• 4:34
  SPEAKER_04: So you probably many people already knew.
• 4:37
  SPEAKER_04: Uh um Carl is a father of uh Larry Page, who is a founder of Google.
• 4:43
  SPEAKER_04: And but of course, uh then we didn't know anything about Google, Google didn't exist, and then but he was very passionate about AI and told us in classroom a lot of stories about AI.
• 4:53
  SPEAKER_04: Then I said, that's really kind of magical, right?
• 4:56
  SPEAKER_04: Then I did two projects under him.
• 4:59
  SPEAKER_04: I said, you know, I really want to do something in at the intersection of AI and the user interface design.
• 5:06
  SPEAKER_04: So like you can see what's why I call the human-centered AI, it's really at the intersection of um human-computer interaction and artificial intelligence.
• 5:15
  SPEAKER_04: So then I decided to pursue a PhD in this particular area.
• 5:20
  SPEAKER_04: So um, obviously, my uh uh thanks to my advisor who really gave me lots of freedom to do research, to explore.
• 5:28
  SPEAKER_04: So my thesis was working on creating uh a really the AI data storyteller, which means it is given a set of data, given a task of the user, and given the user's visual preferences.
• 5:41
  SPEAKER_04: So, for example, some users want to see overview first, then detail, and other users prefer to see the detailed example first and then overview.
• 5:50
  SPEAKER_04: So the program will automatically generate a um data story.
• 5:56
  SPEAKER_04: It's like basically kind of like in which is which consists of a series of animated uh data visualization.
• 6:04
  SPEAKER_04: Interesting.
• 6:04
  SPEAKER_04: So we'll tell the story.
• 6:05
  SPEAKER_04: Yeah, so for example, uh the system, part of the system actually being used by the Columbia Presbyterian Hospital to brief the nurses about the patient status.
• 6:17
  SPEAKER_04: So, for example, we'll tell you the nurse about uh uh what happened during the surgery and what kind of a treatment this particular patient is under, and what the treatments should be given after the, let's say, after the patient uh um uh was came uh after the surgery, right?
• 6:35
  SPEAKER_04: So it's kind of a very interesting telling the story uh to the nurses so nurses can take actions afterwards.
• 6:41
  SPEAKER_01: So you're getting the data to communicate in a way that is more user-friendly and that tells uh a story around whatever it is that's of interest.
• 6:53
  SPEAKER_01: Uh people use that term in in other ways, the data storytelling, talking about data and using data to tell a story.
• 7:00
  SPEAKER_01: So um, so that's very interesting.
• 7:02
  SPEAKER_01: And uh how far along uh did that go?
• 7:05
  SPEAKER_01: And is it uh is it still uh in in use?
• 7:08
  SPEAKER_01: And uh does it still kind of in and and does it uh align with how people talk about data storytelling, right?
• 7:15
  SPEAKER_01: Like there's a guy who we had on the prior podcast, and they they call him the the you know the what is he, the data whisperer or something, but he's he's into data storytelling or stories about data rather than the data telling a story.
• 7:28
  SPEAKER_04: It's uh uh it's basically the data tells the story.
• 7:32
  SPEAKER_04: In our case, it is because uh think about this way, because I knew that's your the knowledge engineering person, my expert, and you can just imagine in that uh you have a pile of data, right?
• 7:44
  SPEAKER_04: But uh in this data, there are some of the golden nuggets, and there's some of the things you don't you don't need it to know.
• 7:50
  SPEAKER_04: It's basically it's just um, I would say this potentially could be noise, right?
• 7:55
  SPEAKER_04: So, how can you uh uh pick out the golden nuggets and thread them together to show a very coherent and meaningful summary to the people who are recipient side, it's very useful.
• 8:08
  SPEAKER_04: So um I did uh know that uh that system was used uh in the cardiac ICU by the nurses for a while.
• 8:16
  SPEAKER_04: Whether it is used by now at the moment, I don't know about that, right?
• 8:20
  SPEAKER_04: So this is actually uh connecting to my next my stage of a career at IBM Watson.
• 8:27
  SPEAKER_04: So after I graduated from Columbia, I joined IBM Watson Research Center.
• 8:31
  SPEAKER_04: So then I said that you know the storyteller we had for the Presbyterian hospital caregivers, it's not very interactive.
• 8:40
  SPEAKER_04: Interactive meaning is this.
• 8:42
  SPEAKER_04: The story, the visual story is interactive.
• 8:44
  SPEAKER_04: You can click on, you can look at the detail, but not interactive in a way the nurses can say, Oh, hold on, I don't want to see this data, I want to see a different set of data.
• 8:55
  SPEAKER_04: Or can you compare this set of data with the yesterday's data, right?
• 8:59
  SPEAKER_04: It couldn't do that.
• 9:00
  SPEAKER_04: So at IVM, my first really significant project is to make the interactive conversational data storyteller.
• 9:08
  SPEAKER_04: So in that case, uh users can interject and can change the trajectory of the story.
• 9:16
  SPEAKER_01: Shape the story, yeah.
• 9:17
  SPEAKER_01: Exactly, can shape the story interesting, yeah.
• 9:20
  SPEAKER_04: Yeah, based on the context, right?
• 9:22
  SPEAKER_04: So I remember we used uh uh we use the data set, which means publicly available data set.
• 9:27
  SPEAKER_04: We can talk about it.
• 9:28
  SPEAKER_04: Is the person was asking, show me the houses in Westchester County.
• 9:33
  SPEAKER_04: I mean, in your case, uh, show me the house about houses in the Boston metropolitan area, and then you would actually go to the database and try to find the information that's uh important, relevant to the users, and then to say, hey, here are the houses, and then actually bend in it and make the summary, and then it's within context, and the person can use natural language to say this.
• 9:57
  SPEAKER_04: Then tell me about the school, maybe doing a narrow down, right?
• 10:02
  SPEAKER_04: Just those houses under 500K uh built after 1970s.
• 10:07
  SPEAKER_01: So you can you're you're not talking about Boston now, yeah.
• 10:11
  SPEAKER_04: Right.
• 10:11
  SPEAKER_01: So basically prices are so high, but anyway, um, no, those are great, those are great points, and so that started to get into natural language understanding story, and then you know, there's this field uh that we talked about psycholinguistics.
• 10:26
  SPEAKER_01: What is that really?
• 10:27
  SPEAKER_04: Tell us very, very I I I love your segue.
• 10:30
  SPEAKER_04: So in the conversational storyteller and uh data exploration, we try to understand the users' preferences because in my field of human-centered AI, one of the very, I would say, uh very important uh study area is user modeling.
• 10:47
  SPEAKER_04: How can you understand users so you can make computer systems adapt to users, right?
• 10:54
  SPEAKER_04: Not another way around.
• 10:55
  SPEAKER_04: So we have I'm my first real be 10 year, maybe 15 years of research is working on understanding users in a task context.
• 11:05
  SPEAKER_04: So what their tasks are, what they're looking for, what their visual preferences are, what their maybe like their verbal preferences, but not on their psychological characteristics.
• 11:17
  SPEAKER_04: So for example, is this person a very open-minded person, or is this person a very careful, cautious person, or is this person very social, or this person very independent, very different.
• 11:29
  SPEAKER_04: Think about it if you know those characteristics, your information presentation, your interaction with this person might be completely different, right?
• 11:39
  SPEAKER_04: So then I started another project uh called the system you at the time, then morphed into the Watson personality insights later on in the commercial world that it's trying to use users' uh communication text to infer people's personality.
• 11:56
  SPEAKER_01: And is that the computational psychology, right?
• 12:00
  SPEAKER_01: When I see the assessments, the tests that say things like, Do you like to eat ice cream or ride a bicycle?
• 12:05
  SPEAKER_01: It sounds you know, they're they're weird questions, but they end up with an assessment that really describes your personality.
• 12:11
  SPEAKER_01: So this is inferring some of that from the interactions, okay?
• 12:15
  SPEAKER_04: So from especially textual communication interaction, right?
• 12:20
  SPEAKER_04: So this is uh this is like your question earlier.
• 12:23
  SPEAKER_04: This is uh actually line of the research behind us called the psycholinguistics.
• 12:28
  SPEAKER_04: So the psychologists, uh linguists have found out that uh even as long as probably back to Darwin time, right?
• 12:35
  SPEAKER_04: So they found out that is uh how people express themselves highly related to their personality.
• 12:42
  SPEAKER_04: So for example, people who are very extra or very social and extroverted, they often probably use this pronouns like a plural pronouns, like a we, right?
• 12:53
  SPEAKER_04: Versus the person who are much more, let's say, keep them to themselves, they're maybe much more likely to refer to I, right?
• 13:01
  SPEAKER_04: So in this case, it is that's kind of like a foundational theory.
• 13:04
  SPEAKER_04: But of course, uh in the past, people have been doing this one by hand, which means experts looking at your writing, looking at your one, trying to figure out who you are, right?
• 13:16
  SPEAKER_04: Like a psychologist.
• 13:18
  SPEAKER_01: Yeah, personas, right.
• 13:19
  SPEAKER_01: Now this is mental models, right?
• 13:21
  SPEAKER_04: Yeah, so Model Mai.
• 13:22
  SPEAKER_04: I would say I would attribute this really to my co-founder, who is a psychologist, right?
• 13:27
  SPEAKER_04: I'm not a psychologist.
• 13:29
  SPEAKER_04: So he's a psychologist and a computer scientist.
• 13:32
  SPEAKER_04: So he said it is, you know, the traditional psychometrics, like you mentioned earlier, is called the item-based of what I we call the test, right?
• 13:42
  SPEAKER_04: They ask you to self-report.
• 13:44
  SPEAKER_04: For example, oh, ask you to on a scale of one to five, scale yourself.
• 13:49
  SPEAKER_04: I love to work with somebody, or I love ballet, or I like to go, I like to outdoor events, something like that, right?
• 13:58
  SPEAKER_04: But it's a self-reporting that has some issues because it's a very subjective.
• 14:03
  SPEAKER_04: So I like ballet, your five and my five may mean completely different meaning.
• 14:07
  SPEAKER_04: And also because of a social desirability bias, as if you are interviewing somebody for a job position, if this person probably will never ever say, I want to work alone.
• 14:19
  SPEAKER_04: Just thinking about it, the working environment, right?
• 14:21
  SPEAKER_04: So in this case, my my co-founder was really brilliant, said this one, you know, using a modern psychometric theory, maybe we can extract evidence from the people's just free text writing and using this what's so-called item-based theory, and they automatically infer people's uh psychographic characteristics, right?
• 14:43
  SPEAKER_04: Like, for example, their passions, their interests, how they handle life's challenges.
• 14:48
  SPEAKER_04: So that's uh so that basically the key of the art of it is how do you identify those evidence and uh what would could be used as evidence to infer those uh psychographic uh uh characteristics.
• 15:02
  SPEAKER_00: So let me ask you a quick question, Michelle, because it's it's interesting, you know, centered around.
• 15:07
  SPEAKER_00: I actually did psychology as well, applied psychology.
• 15:09
  SPEAKER_00: I'm assuming that a lot of the work that you did on Watson then translated over into GOG that you're working on now, right?
• 15:17
  SPEAKER_00: We'll get to that.
• 15:18
  SPEAKER_00: But you know, in terms of a lot of these the psychometric matrices that you guys captured, what did it look like in terms of of accuracy?
• 15:27
  SPEAKER_00: How did you how did you actually get to testing accuracy and then how did you all and then the second part of that question is is bias, right?
• 15:35
  SPEAKER_00: How did you actually you know work against bias in those situations?
• 15:39
  SPEAKER_04: Great question.
• 15:40
  SPEAKER_04: So, first of all, I do wanted to address the the how what type of type of work we transferred from Watson, what do we so I would say it is uh because uh uh in Watson, in IBM, I really think IBM will have a lot of opportunities, so I had accumulated lots of a domain knowledge in my head.
• 15:57
  SPEAKER_04: So that's definitely transferred.
• 15:59
  SPEAKER_04: But the methodology we we have been work using right now in GG we develop, it's completely different from what we have done like in Watson.
• 16:07
  SPEAKER_01: From IBM, nothing.
• 16:09
  SPEAKER_04: No, the reason, no, the reason is no, there's a reason.
• 16:11
  SPEAKER_04: This is a reason I think you probably will ask me about this one from an entrepreneur point of view.
• 16:16
  SPEAKER_04: Because when you develop the approach or method or even algorithm at the companies at IBM, you care much less about the cost, right?
• 16:25
  SPEAKER_04: But when you come out, become your own, I have to pay everything from my bank part company's bank one, I have to be very careful about the cost.
• 16:34
  SPEAKER_04: So that's really motivated us to find completely different approaches than what we had done at IBM, right?
• 16:41
  SPEAKER_04: So then going back to Chris's question about the accuracy, in actually in the psychology world, they don't call it accuracy, they call it two metrics, which is absolutely important to evaluate uh what we have, the things like what we have developed called the psychometric measures.
• 17:00
  SPEAKER_04: Reliability and the validity.
• 17:02
  SPEAKER_04: So reliability means that uh because it's measure, right?
• 17:07
  SPEAKER_04: Any measurement, any instrument you want to use, it has to be measured by reliability.
• 17:13
  SPEAKER_04: What it means is how stable it is, right?
• 17:16
  SPEAKER_04: Like the measure measurement instrument at the weight at the scale, weight scale.
• 17:21
  SPEAKER_04: If today I measure you for 200 pounds, but tomorrow I say you weigh 500 pounds, that's a very bad scale, right?
• 17:28
  SPEAKER_04: Similarly, for psychometrics, reliability, which means that if you are measured this person by, let's say uh 500 words today, uh it's with certain type of uh personality, but tomorrow I measure you another 500 words, you come completely different personality.
• 17:47
  SPEAKER_04: That's problematic, right?
• 17:50
  SPEAKER_04: So, but you have to have a certain, what I call it, uh uh minimum number of evidences, right?
• 17:57
  SPEAKER_04: So reliability, we found that is an earlier one, we found it is uh about a thousand words.
• 18:02
  SPEAKER_04: This also contrasts to IBM.
• 18:04
  SPEAKER_04: IBM, I remember when we published it, is we have to say around 3,000 words in order to derive a stable stable in our case called internal reliability.
• 18:14
  SPEAKER_04: This is one of the very famous measures called the crown box alpha.
• 18:17
  SPEAKER_04: So you have to measure it, have to above 0.75 to 0.8, because humans known are inconsistent.
• 18:25
  SPEAKER_04: So if you take the same test twice, then you may not even be 100% uh same, right?
• 18:32
  SPEAKER_04: You would have reliability around this 0.8, basically.
• 18:35
  SPEAKER_04: That's why.
• 18:36
  SPEAKER_04: So 0.8 is acceptable reliability.
• 18:38
  SPEAKER_04: So well, IBM was 3000 words, our first version was thousand words.
• 18:44
  SPEAKER_04: And now, well, I think one of the papers is going to be coming out by a third-party evaluator from a bunch of professors, psychology professors, maybe 500 words, you will actually reach that uh acceptable reliability.
• 18:58
  SPEAKER_04: Besides reliability, more importantly, it's validity.
• 19:02
  SPEAKER_04: So in psych in psychology, validity, which means it is can you use the inferred characteristics, whatever characteristic you you think it is, right?
• 19:11
  SPEAKER_04: Use that to predict people's real world behavior.
• 19:16
  SPEAKER_04: Because the the really the crux of the research is about the work is about it, is if you learn about the people's characteristics, you want to use that to help people.
• 19:28
  SPEAKER_04: For example, help them be more healthy, help them find a career.
• 19:32
  SPEAKER_04: But if you couldn't demonstrate it, your derived personality or whatever traits you you claim doesn't actually predict the real world behavior or performance.
• 19:43
  SPEAKER_04: It's useless, right?
• 19:44
  SPEAKER_01: So right now, overall, self-declared, it's like self-declared expertise.
• 19:49
  SPEAKER_04: Exactly, right, right.
• 19:50
  SPEAKER_04: So you think that very useful.
• 19:51
  SPEAKER_04: So you you know the personal test.
• 19:53
  SPEAKER_00: That's why I asked the bias question, too.
• 19:55
  SPEAKER_00: Exactly.
• 19:55
  SPEAKER_04: So the personality test validity, yeah.
• 19:58
  SPEAKER_04: Yeah, the traditional personality tests, why people still use it?
• 20:02
  SPEAKER_04: It's because it proves it has certain validity, right?
• 20:07
  SPEAKER_04: So people do that when you say this.
• 20:09
  SPEAKER_04: Like, for example, you're welcome to take a look at this.
• 20:11
  SPEAKER_04: So one of a very famous paper called the power of personality, and it shows that a certain personality can predict some people's lifespan was what could be seven years, average, longer than other people.
• 20:25
  SPEAKER_04: That's a very strong statement, right?
• 20:27
  SPEAKER_04: And some people are much more successful academically and at certain job roles.
• 20:33
  SPEAKER_04: So similarly, we want to do the same thing.
• 20:35
  SPEAKER_04: I'm not gonna reveal them a lot of detail because it's papers coming out, but the psychologists have to prove the incremental virility.
• 20:43
  SPEAKER_04: Incremental validity is a very important concept, which means it is you have the both tests, right?
• 20:49
  SPEAKER_04: Uh traditional tests and our test, and you found out that the the inferred treats can be used to predict certain real-world user behavior, okay, which cannot be predicted or be less predicted by the traditional tests.
• 21:09
  SPEAKER_01: Let me let me uh pause one moment um because we're about halfway through to our time.
• 21:14
  SPEAKER_01: I wanted to remind our listeners we're talking with uh Michelle Zhao, who is uh an expert in digital assistance, and we're talking about a particular angle around computational uh psychology.
• 21:26
  SPEAKER_01: Chris, you were you were kind of asking other questions.
• 21:28
  SPEAKER_00: Yeah, I was I was gonna jump in, um, you know, in and basically, you know, Michelle, I'd love to get a perspective to moving from you know the Watson and I I I I can definitely appreciate the notion of going from pure research to maybe it'll make it into products, but really your goal is just pure research to a startup, which is a dramatic contrast between the two, because you know, you talked about yourself use of proceeds.
• 21:58
  SPEAKER_00: You don't have to carry about care about.
• 22:00
  SPEAKER_00: that, you know, when you're in the big uh the big engine.
• 22:03
  SPEAKER_00: But when you get into use of proceeds and this whole notion of of wanting to, you know, wanting to actually build a product, how how and why did you move from Watson and the comfort of research into a product company that's basically funded and started by you.
• 22:21
  SPEAKER_00: But why go from a big company to a startup?
• 22:23
  SPEAKER_00: What was the motivation?
• 22:24
  SPEAKER_04: Great question.
• 22:25
  SPEAKER_04: To be honest, I never thought about to work in a startup.
• 22:30
  SPEAKER_04: I always think about I will be the research for life or maybe go to university and teaching or something like that.
• 22:37
  SPEAKER_04: So um uh my I would say it's really my co-founder kind of persuaded me to into it right for two reasons.
• 22:47
  SPEAKER_04: First one, um because of the success actually of a Waterson personality inside at IBM, many people say well God, this is a really exciting research area we publish a lot, right?
• 22:58
  SPEAKER_04: But the way you see it is it's kind of like a pity if we just remaining into it I'm not talking about the particular method.
• 23:06
  SPEAKER_04: It's about this whole area, right?
• 23:08
  SPEAKER_04: To be able to deeply understand people.
• 23:12
  SPEAKER_04: And if nobody works on it further and if this one just remains to be in research not to benefit especially when I'm talking about general public.
• 23:22
  SPEAKER_04: I'm not talking about a big company who can pay IBM to use this product right I'm talking about anybody who just say hey you know I want a career advisor I couldn't afford right now.
• 23:32
  SPEAKER_04: I remember in our area people pay$25,000 for their kids to figure out what major to apply to college right that's why so I want to really democratize the use of this very cutting edge type of technology.
• 23:46
  SPEAKER_04: Number two it is you really have a hard time to do it as a very big company because a lot of concerns about it you don't have the really freedom right because you want to move fast as well.
• 23:57
  SPEAKER_04: That's why yeah right so my co-founder of course uh then I said it is so we have to build it from scratch yes right so that's why I would say again I kind of just say very naively to say we didn't have even an idea what product will be to be honest Chris we came out we just said this we want to to make sure that people can uh can actually benefit from the cutting edge technology like this no idea what it is so it took a lot took us a long time to figure out what to do.
• 24:27
  SPEAKER_00: Nice well now you're in the middle of it right and um and obviously you know from from GG you know it's really focused on and correct me where I'm wrong but like you said you know university digital systems as well as um you know HR you know type environments and things like that.
• 24:49
  SPEAKER_00: And I'd I'd love to get a take because you know part of this, you know, um having spent a lot of time building and uh working on use cases for Amazon Lex, right?
• 25:00
  SPEAKER_00: That's one of my backgrounds is you know the notion of you know that a lot of the AI centered around you know the short form intent based frameworks is really driven by outcomes, right?
• 25:15
  SPEAKER_00: An outcome basis.
• 25:16
  SPEAKER_00: This was also some of the some of the form the forming of what my bias conversation or you know comment and question was all about is you put it into a university setting that university is looking for obviously you know there's there's specific metrics that they're being goaled on and you know and let's say that I'm a student and I'm looking for shoot I don't know right I don't know what I what I want to be when I grow up and I don't know what I'm looking for in classes that resonate with me and my personality and all that kind of stuff.
• 25:50
  SPEAKER_00: And yet what I see is some recommendations coming from the university saying you know what Chris you should go into computer science or whatever.
• 26:01
  SPEAKER_00: Now is that a true basis on my personality type and my ability to be successful or is it on the bias that the con that the organiz that the university needs to put butts in seats for computer science right that's another in between yeah very yeah very good question.
• 26:20
  SPEAKER_04: Actually um uh we have been working with universities quite a bit right and actually you might be surprised that to your or to you to the opposite that you said universities actually really want students can get you the program who are the best fit who will be the better program would be the best fit for the students you know why because I remember I spoke with the dean of the university business school right it's in our local university business school so he was saying that it is the tools like a GG would be so useful not just for students even for the university to better manage their resources right so thinking about this one students uh especially younger age they don't know what their who what's what would be great for them right so they didn't have uh uh maybe the privilege didn't have the uh I think you cannot not every student can afford a student an advisor personal advisor right so they get you into a university program by accident by just hearing other people say I went to this university with this dean then I I I gave a talk I said this could you tell me which major you are in 80% of people raise their hand saying they're in accounting then I ask do you love accounting?
• 27:41
  SPEAKER_04: And why did you get into accounting?
• 27:43
  SPEAKER_04: They said oh because my uncle said so problem is this then accounting classes are completely full the students who couldn't take the classes have to take six years versus four years.
• 27:54
  SPEAKER_04: But some of the students are brilliant for being a marketing student or being an entrepreneur program they do not they didn't know right and the university was scrambling to say we have to go hire more accounting faculty members but hey uh maybe I'm um because I I was born in China I grew up in China I I did have this influence on my Chinese uh philosophy or maybe culture we believe that everybody is born with the gift right not everybody needs to be a computer scientist and not everybody needs to be accountant there's so many abilities they have unique yeah so you're identifying those unique abilities through these cycles right to have the best match right so that's that's a university one which to my surprise said oh maybe everybody should go to computer science because they make most money no no no universities are very I would say prudent are very wise to see that they just want students to be successful.
• 28:53
  SPEAKER_04: Exactly I think you know this is this is where we get into you know the the ethics behind you know these types of recommendation engines that are actually critical and where transparency you know drives accountability I I think Chris I really love what you just said transparency really drives uh uh responsibility as well right we I often say it is uh with the great power comes with the great uh responsibility and uh our AI now has this power and we want to it be used in a very responsible way.
• 29:26
  SPEAKER_01: Right.
• 29:26
  SPEAKER_04: Let me ask you this what have the lessons learned been from your startup so you you started up you know not necessarily knowing exactly what you wanted and what's what's that evolution been like I have to say one lesson only one lesson probably for us for every entrepreneur you really really have to build a product that can help people I'm gonna use Chris the word achieve achieve their outcomes not your outcomes right just a very simple story to you so when we created the GG the very first time we wanted to GG to be an interviewer because in this case when you interview let's say candidates interview students you can help them assess their fitness assess their capability and actually their characteristics right but first of all it's not we have to create the interviewer for every person who wants to try have to customize it second of all there's no results can be easily visualized the view they have to download the whole big uh spreadsheet so our we used to be so confident to say so what a survey monkey gave you that one and the Quantrix gave you that one why should we take energy to go into get created the end-to-end solution right you can see the results you can see the magic quadrant of the people where the fit where the risk is you know what I had a very very honest feedback to me from my dear friend the Saint Michelle you know your tool sounds very interesting sounds very useful but in the current state it doesn't help me a lot doesn't help me at all it just adds burden to my work because I have to ask you to customize the question if I don't like the question.
• 31:14
  SPEAKER_04: I have to you customize the AI if I don't like what the AI response is then I have to analyze the free text responses from all these people it just adds more burden to me.
• 31:26
  SPEAKER_04: Why?
• 31:27
  SPEAKER_04: Why should I use your product so that's really I would say wake us up you have to build something you can't be say 80% their way has to be 100% right so I never figured out because I was reading before I became entrepreneur because of course we're we're geeks right we're scientists we always kind of read try to read from a reading to to learn and there's a lean startup right I remember I said it is MVP do it if it's not quite work and do it.
• 31:55
  SPEAKER_04: You know at least from our experience something not quite work nobody wants it.
• 32:00
  SPEAKER_03: Right.
• 32:01
  SPEAKER_04: Even it's free nobody wants to try because it's wasting their time.
• 32:07
  SPEAKER_00: Right?
• 32:08
  SPEAKER_00: So let me let me ask you this on that thread and this is going to be near and dear to um to Seth's you know heart passion is my reason for existence to what's that my reason for existence exactly exactly um your other one and only someone um when you when you get into this with organizations Michelle what you know what does it look like you know centered around because you know and I'll I'll preface the question quickly most organizations don't even uh we either don't know the data they have or they have a subset of data that they want to utilize and um and or they think they have what they want to base things on but they you know that it's not in you know it's not inclusive.
• 32:54
  SPEAKER_00: So how do you work on data governance and discovery and taxonomy?
• 32:59
  SPEAKER_00: What does that look like when you guys go and work with an organization?
• 33:02
  SPEAKER_04: So actually this is a very good question, right?
• 33:05
  SPEAKER_04: So for example you're right uh when we're working with universities or even uh uh HR department uh they often ask they often say it is we have this data we have this data and we always go try to say what kind of a data is available so for example even the FAQ data right so just I just use the simplest example if you look at the FAQ data sometimes FAQ the answer is very very long right because that's made for the website consumption.
• 33:35
  SPEAKER_04: So you can read it but you just kind of completely move it to the chat people will be very very annoyed because it's such a long one have to scroll so much.
• 33:46
  SPEAKER_01: Remember the chat window it's so small normally on your little corner so we always kind of ask this uh first one to figure out what content is available second one it's uh format of the content right so uh recently um uh we have been actually uh uh in our team we've been discussing about the from a content point of view if you have the content uh how can you best present it basically it's a form and the content right so the maybe tool like a GPT uh chat GPT already gives you that kind of content but you have to decide how to best present it in the context so there's a kind of a multiple oh I wouldn't say multi-step process but often people probably are unaware or don't realize there's a lot of work needs to be put in place for that right right the knowledge sources uh need some degree of curation at least I still am coming across vendors that say you know our AI does it all our AI you know magically I know and it and it's it's like seriously guys come on you know you can you know and uh it's frustrating but but talk a little bit about general language models such as uh GPT 3 and then chat uh gpt I have not investigated chat GPT I've heard interesting things about it but I'm also seeing you know I think the chat um uh uh the chat space is getting very interesting from the perspective of misconceptions I just read an article how chat is going to help with drug discovery it's like chat is a channel right how are you talking about drug discovery just in terms of a channel it's a very interesting but but the idea of people saying well now we have uh GPT3 or we have chat uh GPT how is that impacting I look at it as a generalized language model is going to be okay for common tasks but you're still going to need a specific model for terminology and an industry and an organization so first of all talk but talk to for people maybe it's less familiar talk about GPT3 and then talk about chat GPT.
• 36:03
  SPEAKER_04: So GPT3 it is uh a large language model right so what happened is uh it uh oh it gets uh basically a lot of training data if you will right and from the training data it tries to synthesize generates different types of answers I think that right now the chat GPT is still uh built on top of the GPT3.
• 36:28
  SPEAKER_04: It just has the interface for you to access the information that is already kind of uh even be trained and will be synthesized right from the underlying GPT 3 model.
• 36:41
  SPEAKER_04: So recently I have been actually uh interacting with the chat GPT uh it's interesting uh I'm very impressed by some of the capabilities right so for example uh yesterday I was trying one of the queries I would say uh persuade a person who is very family oriented to quit smoking I did the comparison I also persuade a person who is very self-sufficient independent to quit smoking so the synthesizes of the two persuasive if you will statements are very different right they put on a but hey then people said then chat GPT already did all this one why would companies like GG still exist but uh here what I said I put into the query to say family oriented person I put in a query to say self-sufficient independent person where does such knowledge will come from how did you know the person is one so like you said first one is in order to run chat GPT in a reasonable way you need context you need parameters so the in the conversations like the GG has is to gather that context to help and also that as you said as I mentioned earlier the presentation that you you can't just directly use the in conversation you still have to edit a little bit but it helps the conversation designer helps whoever manage the conversation really saves a tremendous amount of time is number one right and the second part of it is you still need the context that like you said the task context.
• 38:25
  SPEAKER_04: So for example I asked something to say please compare because recently I've been doing lots of gardening please compare David in David Austin Rose's golden celebration versus teasing Georgia.
• 38:37
  SPEAKER_04: It has no idea what it is right it just said it is I don't know anything about plants of course I know stats you have been working with you doing a lot of consultant technology engineering with enterprises so it has to be proprietary data you have to train your proprietary proprietary models as well so those things which means it is I would say uh chat GPT really gives you a glimpse what future could hold but there's still a lot of work a lot of training still needs to be done by the specific entity specific uh organization and by companies like us to provide that kind of context so you think there'll still be a future for knowledge management and knowledge engineering absolutely actually we'll make the we'll make the line of the work you do knowledge management even more important right because you would uh instruct you would guide the tools like GPT3 to be much more targeted to be much more accurate right right no that's a great that's a great point that's application people talk about the uh proprietary models and also application layer right when you put application layer in terms of like for example like a financial like finance right versus healthcare the wording the type of presentation are completely different general model doesn't know that context you have to put the context into it but you can leverage what it can do right so let's talk about uh the coming disruption we know these models are getting better we know that we're automating more things that humans do there will be disruption in the marketplace by AI what do you think um first of all what new jobs will be created to support cognitive AI and then how will AI assistance help with job disruption I would say the type of job new job would be you know we have many main domain experts right like for example uh domain experts who knows about the marketing who knows about recruitment who knows about the sales those people will be supercharged by what I call it the current AI like the uh computational psychology with the GG's doing like a chat GPT one to supercharge them so that they can actually do their job maybe 100 times thousand times faster and better but uh they were the brain so for example like an example like I said today right and you want to for example if you are the head of the smoking program somehow like that I they would devise a strategy they want to design devise what I call the business rules but uh the execution would be AI assistants so almost like you everybody could have an army of AI assistants right so in a GG's ambition is we want to become the platform the manufacturer of any AI beans I don't call it AI assistant anymore we call it AI beans you can come in to create your own custom AI beans right either be your companion could be your girlfriend could be your boyfriend and it could be your just friend could be your assistant as well right so that's the one we want it's interesting because I'm starting to see this um you know the notion of a digital assistant but in the form of like you said an EA and or you know like the application is really really rich especially amongst like small to medium businesses that need you know that that need to have the ability to have an EA type of perspective but don't can't afford it.
• 42:22
  SPEAKER_00: You know what I mean?
• 42:23
  SPEAKER_00: Exactly exactly you could definitely attach a text based assistant and a voice-based assistant into the same conversational structure and then still have the ability to you know to take action to um you know to automate you know specific tasks and things like that that you know are very generic and germane to let's say you know having you know somebody that can like an assistant that can actually really help them.
• 42:50
  SPEAKER_00: You know what I mean?
• 42:50
  SPEAKER_00: So I'm starting to see those types of things happening you know a lot more frequently than we have in the past.
• 42:56
  SPEAKER_00: But I'd love to get an idea too like um you know how what you guys are doing is is ahead of the curve from everything else.
• 43:07
  SPEAKER_00: What does that look like?
• 43:08
  SPEAKER_00: Don't I mean don't don't share your IP right but but what are the things that that you know set you guys apart?
• 43:13
  SPEAKER_04: So you are absolutely right that those uh uh assistant uh digital assistant AI assistant right can help small medium businesses so just thinking about when those uh businesses hire a human assistant what the requirements are of course uh understanding English is a must otherwise this person cannot interact with the customers or partners anybody right but uh it's not sufficient correct they want this person to be also has a certain level of what we call the social emotional to be able to know how to work with people how to interact with people especially when they don't have the knowledge right that's what uh Gigi sets it apart from the rest of them is our goal is to teach the AI assistants or digital assistants, what we call the advanced human skills.
• 44:06
  SPEAKER_04: Active listening, and even they don't know how to answer the question, but do it in a very uh kind of uh with the social-emotional, actually, uh uh how to say statements, right?
• 44:18
  SPEAKER_04: Another one is uh reading between the lines, really understand uh what they want, what they're like, right?
• 44:26
  SPEAKER_04: Once you do that, thinking about it, uh, and um uh Chris and says that you have your own digital AI assistant, and uh let's say I'm reaching out to you, it doesn't know who I am yet, but then after I converse with and I want to tell you guys to say, you know, I today I spoke with Michelle.
• 44:42
  SPEAKER_04: Michelle is a very um uh enthusiastic, uh very open-minded person about the technology, and uh he wants to talk to you guys.
• 44:51
  SPEAKER_04: Then you can decide is this the right fit for your consulting or not, right?
• 44:56
  SPEAKER_04: So that's how your AI assistant can really take a step further from those mundane, I call it transactional operations.
• 45:06
  SPEAKER_04: Okay, take a message, make appointment.
• 45:09
  SPEAKER_04: Yes, it needs to do that, but the more to that it is, for example, if you guys are away, if I ask it is so tell me more about the status operation, what kind of businesses he likes to do it?
• 45:21
  SPEAKER_04: Well, what kind of thing?
• 45:22
  SPEAKER_04: Then it gives me the idea whether this is something I want to continue engage with, right?
• 45:28
  SPEAKER_04: So that's what's really get to the that's why we also talk about it is uh this empathetic automation of a high-touch services which aren't possible and before.
• 45:42
  SPEAKER_00: I love the the the personal assistant type of scenario because now what we're talking about is it understanding, you know, me at scale, right?
• 45:52
  SPEAKER_00: Or understanding the single, the singular at scale.
• 45:56
  SPEAKER_02: Yeah.
• 45:57
  SPEAKER_00: Exactly.
• 45:57
  SPEAKER_00: And I and so part of that too is you know, there was use cases that that I wanted to actually work on, which I'm glad you're solving now, where you, you know, a new person that's interacting with a giant organization like a like a new um employer or a new university or something like that, needs that personal assistant to do anything from like, hey, uh, you know, like maybe it's on my mobile, tell me where this class is, and then boom, you get directions.
• 46:30
  SPEAKER_00: Or tell me what books I need to order or when my bills are due, or you know, things like tell me how to not, you know, gain the freshman 15, you know, in terms of weight.
• 46:41
  SPEAKER_00: What can I do?
• 46:42
  SPEAKER_00: You know, and it's that that interaction, you know, at scale, but it understanding, you know, who me, who I am, what my needs are.
• 46:50
  SPEAKER_04: Yeah, and you know, maybe I also encourage you to stay on course, right?
• 46:54
  SPEAKER_04: Okay, today I was trying to also do that.
• 46:56
  SPEAKER_04: You see, it is uh um one of the things is universities and uh uh any academic programs are worried about it, is especially from an online learning uh form, and it's very it's a lonely journey, right?
• 47:08
  SPEAKER_04: People learn at their home.
• 47:10
  SPEAKER_04: I know some people have a most people have a part part-time or full-time job.
• 47:14
  SPEAKER_04: Stay on course, adhere to the learning process.
• 47:18
  SPEAKER_04: Uh, it's a tremendous challenge, right?
• 47:20
  SPEAKER_04: With so many distractions.
• 47:22
  SPEAKER_04: So uh I think the assistant like this, we call the learning companion or learning body, really not just help you on the mundane operational, transactional um items, but really actually be your, I would say, emotional partner too, right?
• 47:37
  SPEAKER_04: Encourage you, it's maybe they know you, right?
• 47:40
  SPEAKER_04: And uh hey, encourage you to stay on course.
• 47:43
  SPEAKER_01: Absolutely.
• 47:44
  SPEAKER_01: Well, listen, we've come to the end of our time today.
• 47:47
  SPEAKER_01: It's been a real pleasure talking to you, Michelle.
• 47:50
  SPEAKER_01: Uh, I've really enjoyed it and fascinating, fascinating topics, and really uh looking forward to following more of what uh Juji is doing.
• 47:58
  SPEAKER_01: Uh so thank you again, and uh we'll look forward to uh perhaps having you back on so that we can continue this conversation.
• 48:06
  SPEAKER_01: But this has been fabulous.
• 48:08
  SPEAKER_04: Thank you, Sat.
• 48:09
  SPEAKER_04: Thank you, Chris, for the great questions, and thank you for having me.
• 48:12
  SPEAKER_00: Thanks, Michelle.
• 48:13
  SPEAKER_00: It's our pleasure and look forward to uh working with you in the future.
• 48:17
  SPEAKER_00: Good luck.
• 48:18
  SPEAKER_00: Thank you.
• 48:22
  SPEAKER_05: Thanks for listening to the Early AI podcast with your hosts, Seth Early and Chris Featherstone.
• 48:28
  SPEAKER_05: We hope you enjoyed the program and took away some ideas that you can start to implement in your organization right now.
• 48:34
  SPEAKER_05: If you're stuck and need help, check out www.early.com for information about early information science consulting services.
• 48:43
  SPEAKER_05: And of course, check the show notes for links to any resources we mentioned today.
• 48:47
  SPEAKER_05: That's all for now.
• 48:48
  SPEAKER_05: See you next time.