Latest / The Adults In The AI Room / The Earley AI Podcast with Seth Earley - Episode #53- Insights into Data Science and AI Validation from Expert Bartek Roszak

Transcript

The Earley AI Podcast with Seth Earley - Episode #53- Insights into Data Science and AI Validation from Expert Bartek Roszak · The Adults In The AI Room

• 0:00
  SPEAKER_02: Well, good morning, good afternoon, good evening.
• 0:02
  SPEAKER_02: Whenever you're listening to this, uh, welcome to uh the show.
• 0:05
  SPEAKER_02: My name is uh Seth Ruby.
• 0:07
  SPEAKER_03: And I'm Chris Featherstone.
• 0:08
  SPEAKER_02: And what I'd like to do is introduce our guests.
• 0:11
  SPEAKER_02: We're really excited, uh, and we're gonna talk about many of the misconceptions around uh AI and general generative AI.
• 0:17
  SPEAKER_02: We always talk about misconceptions.
• 0:19
  SPEAKER_02: We're gonna talk about retrieval augmented generation, which of course is the thing that people think is you know the next best thing.
• 0:26
  SPEAKER_02: Um, and uh, of course, there's there's good reason to think about that.
• 0:29
  SPEAKER_02: Uh, we'll talk about um some of the nuances of REG.
• 0:33
  SPEAKER_02: Our guest today is a round expert in artificial intelligence and data science.
• 0:38
  SPEAKER_02: He has a unique background uh that includes uh economics.
• 0:41
  SPEAKER_02: He started his career as an equity trader on the New York Stock Exchange and positioned a transition from using Excel to do his data-driven decision making to mastering advanced data science techniques with Python and R.
• 0:54
  SPEAKER_02: He has a deep understanding of statistics and advanced mathematics, and that coupled with his practical experience really uh propelled him into the world of AI because that was an ideal uh background for that.
• 1:06
  SPEAKER_02: Currently, he is head of AI at STX Next, where he leads AI strategy in implementation for a range of innovative products, uh projects.
• 1:14
  SPEAKER_02: Bartek, Rosak, uh, welcome to the show.
• 1:18
  SPEAKER_01: Yeah, hi Seth, hi Chris.
• 1:20
  SPEAKER_01: Uh thank you for the introduction.
• 1:22
  SPEAKER_01: It's great to be here.
• 1:23
  SPEAKER_03: So, Bartek, were all of those, it seems like you had, like as an equity trader, building tools to help you trade faster, better, cheaper, stronger, whatever that is for you personally, or was it for the companies you're working for?
• 1:37
  SPEAKER_01: So, like I would say it both.
• 1:39
  SPEAKER_01: Of course, I did it for for like I started as a trader for a propriety trading company during my like Excel time.
• 1:47
  SPEAKER_01: And after like I started doing data science, deep learning, and stuff like that.
• 1:52
  SPEAKER_01: I also worked for a hedge fund where I built um um autonomous traders, but also during my free time, I like to to experiment with uh trading.
• 2:01
  SPEAKER_01: Of course, I have my own trading strategies rather long terms, but and and I use a lot of data science there.
• 2:06
  SPEAKER_03: But yeah, maybe Seth, we should get an offline tutorial from Bartek on our trading strategies and I'm in index funds.
• 2:14
  SPEAKER_02: That's it.
• 2:14
  SPEAKER_02: I'm in index funds, long-term buy hold.
• 2:17
  SPEAKER_02: I'm I'm not a day trader.
• 2:19
  SPEAKER_01: I would have to go on a pretty good strategy that is really hard to beat, actually.
• 2:24
  SPEAKER_02: That were the Wall Street Journal has people throwing every year, they do the darts.
• 2:29
  SPEAKER_02: Have you heard about that?
• 2:30
  SPEAKER_02: Completely random, and the random dart throwing uh typically outperforms uh the market.
• 2:36
  SPEAKER_02: I who can say well, all right, or out outperforms one of the more advanced um, you know, gurus and and uh star traders out there.
• 2:44
  SPEAKER_02: But let's uh let's talk about the space that we're in, the Gen AI space, the AI space, and talk about what are you seeing in terms of your projects, your initiatives when you talk to executives, either on the technology or the business side, what are the things that people don't understand or what do they get wrong?
• 3:00
  SPEAKER_02: And you know, I know that's kind of evolving, right, as the industry matures, as people become more aware, but give me a give me your thoughts on on what the common misconceptions are.
• 3:10
  SPEAKER_01: So I think that in the N AI, especially in Gen AI, we have this common thing with uh uh with the new things that are really hyped.
• 3:19
  SPEAKER_01: So so, like one and a half year ago, we we heard that prompt engineering is is the next big thing.
• 3:26
  SPEAKER_01: Then we heard that uh rack is the next next big thing.
• 3:30
  SPEAKER_01: I think that I hear now that uh multi-agent uh approach is is the next big thing, right?
• 3:36
  SPEAKER_01: So so so I'm curious what would be the next big thing after that.
• 3:40
  SPEAKER_02: But yeah, independently rag um approaches with prompt engineering, yeah.
• 3:46
  SPEAKER_01: Of course, this is this is all actually the same stuff.
• 3:49
  SPEAKER_01: It's all the same stuff because you know we start uh achieving like some better results with prompt engineering, but it was not enough.
• 3:56
  SPEAKER_01: So we start doing rack.
• 3:58
  SPEAKER_01: After we do rack, we we we we we see the the limitations, so we start doing multi-agents and stuff like that.
• 4:05
  SPEAKER_01: So we still it's like a normal process of of development, I would say, right?
• 4:09
  SPEAKER_01: So basically, in terms of like what I see on the market right now, so so so we have this this huge hype.
• 4:16
  SPEAKER_01: A lot of companies want to do AI.
• 4:18
  SPEAKER_01: Some of them like are are major in doing genai, some of them just started, and and also a lot of companies started AI projects that failed, and they they don't want to now to to to invest more of them, more of our money to to to projects like that.
• 4:33
  SPEAKER_01: But there's also a lot of great success in in applying AI generative AI rack uh and and multi-agent approach, and and and we see more and more companies that want to want to do the same, right?
• 4:46
  SPEAKER_03: So I got a quick question too, because we usually ask, and Seth usually starts out the misconceptions you know for AI and stuff.
• 4:53
  SPEAKER_03: And I I I love the fact that you come from a deep data science background.
• 4:57
  SPEAKER_03: Where do you think, you know, kind of going up the stream a little bit, where do the data science teams miss in a lot of this?
• 5:05
  SPEAKER_03: Or do you feel like they're yeah?
• 5:07
  SPEAKER_03: Yeah, sorry, go ahead, Bartek.
• 5:09
  SPEAKER_03: Go ahead.
• 5:09
  SPEAKER_01: Yeah, the huge gap that I that I see now is it's quality assurance, right?
• 5:14
  SPEAKER_01: So so basically in normal like software development, we have QA teams, and it's like pretty obvious that we need QA team that is separated from the engineering team to check how good, uh what is the quality of their work and stuff like that.
• 5:30
  SPEAKER_01: Uh, but actually, we don't have good practices around AI, like around QA and AI.
• 5:36
  SPEAKER_01: And I've seen a lot of AI projects that failed because they didn't have proper QA, because people who developed models were also the same guys who were evaluating the models, validating the models, and deploying models on production, right?
• 5:51
  SPEAKER_01: So so this is this is something that I believe is missing right right now in AI and it would and will change in the future.
• 5:58
  SPEAKER_01: Even right now, like after I joined to STX next like one year ago or something, the first thing I started to to do in I started uh meet meetings with QA team and saying to them that okay guys, you are great in software development QA, but you need to be good also in AI QA because this is the next big thing in QA, I believe.
• 6:19
  SPEAKER_01: And yeah, and and and now I start seeing like job offers, like be machine learning QA engineer or AI validator.
• 6:27
  SPEAKER_02: Yeah, so you're talking about quality assurance, right?
• 6:30
  SPEAKER_02: When you say quality assurance, and when you define quality assurance uh in AI or in data, you know, tell me more what you mean by that.
• 6:41
  SPEAKER_02: Because you know, I read a post the other day by a uh chief information officer or chief data officer, and he went on and on about quality data, data quality is important, data quality, data quality, data quality, right?
• 6:55
  SPEAKER_02: And yet it was very difficult to kind of understand exactly what data he was talking about.
• 7:01
  SPEAKER_02: And he talked about training data.
• 7:03
  SPEAKER_02: Well, training data can be interpreted in a lot of different ways.
• 7:07
  SPEAKER_02: So when you talk about quality assurance and and and data quality and quality assurance of models and performance, tell me more what you mean specifically, and then how does that fit in with the different uh data sources?
• 7:20
  SPEAKER_02: So, what do people need to do?
• 7:22
  SPEAKER_01: Yeah, so that's actually a really good question, is because there's a at every stage of AI development, we need some QA and data, it's a first first place where we need to apply some QA, and and there is like a bunch of techniques, it all depends on the use case, but it could be, for example, checking distribution of new data, right?
• 7:43
  SPEAKER_01: So if the distribution is the same that us we have and and our history date, this is one thing that we can check, like missing data, checking missing data, how many missing data do we have, and stuff like that.
• 7:54
  SPEAKER_01: Because if we retrieve some real-time data, there also there always could be some some um, I don't know, some some interruptions, some bugs, and and some changes in the data, and we need to be alerted about it, we need to switch off models and stuff like that.
• 8:09
  SPEAKER_01: So, this is like uh QA data part.
• 8:11
  SPEAKER_01: Is of course it's it is much bigger, but just to give you some example, then we have uh then we develop models, assuming that we have high quality data, history data, historical data as well as real-time data, it's it's good quality.
• 8:24
  SPEAKER_01: Then we develop models, we can develop models with it.
• 8:26
  SPEAKER_01: And after we deploy the models, we like even before we deploy the models, we need to validate this model.
• 8:32
  SPEAKER_01: So there's a lot of things that can happen if you put uh data scientists to to develop new models.
• 8:38
  SPEAKER_01: They they they can by accident miss a lot of things.
• 8:41
  SPEAKER_01: Like they can they can miss some data leakage, they can they can like overfit the validation set.
• 8:47
  SPEAKER_01: They they can do a lot of things, and and even extremely experienced data scientists, deep learning, engineers, researchers can do it.
• 8:54
  SPEAKER_01: I I've seen it many times.
• 8:55
  SPEAKER_01: Did it many did mistakes like this many times, even though I was like extremely aware of it, I would say.
• 9:02
  SPEAKER_01: So validation models before we we run on production is is is another part of QA.
• 9:08
  SPEAKER_01: And the last part of QA is what we do after after deployment stage.
• 9:12
  SPEAKER_01: So our model, we we are pretty sure that our model has high quality data, it was trained well because we we check it everything, and now we deploy it on production and it should work well, right?
• 9:22
  SPEAKER_01: But maybe there is something missing, and we need to monitor that.
• 9:26
  SPEAKER_01: The most common thing that we need to monitor is the output distribution.
• 9:30
  SPEAKER_01: Like there's of course more of it, but but the most common is if we have, for example, prediction, we if we have like classifier, we predict multiple classes.
• 9:38
  SPEAKER_01: We should like put attention to what kind of classes we predict the most often, if if this is like different than it was uh during production, the during uh training phase and stuff like that.
• 9:49
  SPEAKER_01: So this is also like part of quality assurance, right?
• 9:53
  SPEAKER_02: So when you are talking about building a model, training a model, are you actually saying that you're retraining a foundational model?
• 10:01
  SPEAKER_02: You're not building a model from scratch, or are you talking about training as in what the model needs to access in terms of data sources?
• 10:10
  SPEAKER_02: So, how are you defining because training can be defined in a lot of different ways, training data?
• 10:14
  SPEAKER_01: Well what I said is it was more general approach to training machine learning models, not only like generative AI models, but any kind of machine learning models, I would say.
• 10:24
  SPEAKER_01: So simply every time we use data to train models to make predictions and production, it was like general approach.
• 10:31
  SPEAKER_01: But in terms of generative AI, the the things are getting more and more complicated.
• 10:35
  SPEAKER_01: Of course, we don't need to train the model.
• 10:37
  SPEAKER_01: Maybe we want to fine-tune model, but but this is not like uh this is a rather rare case.
• 10:43
  SPEAKER_01: But at the end, we can have like really hard times to evaluate quality of our solution.
• 10:49
  SPEAKER_01: If that depends, it could be RAC, it could be just prompt engineering, it could be a multi-agent approach, but it's really hard to evaluate how accurate our solution is, right?
• 11:02
  SPEAKER_01: Because if we would like to have a system that is able to answer questions about our knowledge base, we have, I don't know, we have some information about our company, internal policies, and stuff like that, and we want to create a rack system that that we can query with some information.
• 11:17
  SPEAKER_01: And and how we can check, how we can evaluate that our system is answering in the correct way.
• 11:25
  SPEAKER_01: We can create like evaluation set, correct questions and answers, but it's not obvious.
• 11:31
  SPEAKER_01: This is like we cannot like perfectly uh say that we have 80% accuracy, 90% accuracy.
• 11:37
  SPEAKER_01: Sometimes model can can give you the right answer, but it will be different than than the golden standard.
• 11:45
  SPEAKER_02: So we have golden use cases where we know the answers, but those are narratives, and the narratives can change.
• 11:51
  SPEAKER_02: And so then we use a we can use a large language model to compare those narratives and to provide some uh uh prediction of accuracy work.
• 11:59
  SPEAKER_01: Yeah, we we we we can do it like that, yeah.
• 12:02
  SPEAKER_02: So you have to kind of or or the human needs to look at them and say these answers are equivalent, right?
• 12:07
  SPEAKER_01: Yeah, actually, we we my personal view is that we don't have like other good strategy than just manually check the answers.
• 12:15
  SPEAKER_01: And because we would like to check the number that is statistically significant, it's really hard to evaluate model after each change, right?
• 12:25
  SPEAKER_01: So so so we of course we we use some automatic system, we use LLM to check LLM answers and stuff like that, but yeah, this is not like I would say perfect steps.
• 12:34
  SPEAKER_02: Yeah, we need curation, you need human in the loop for various validations.
• 12:39
  SPEAKER_02: Now, going back to kind of the misconceptions, one of the things we talked about on our prep call was that people expect it to work.
• 12:46
  SPEAKER_02: Like, tell me more about what you've experienced when it when it comes to Gen AI.
• 12:50
  SPEAKER_02: And and I think there's a little there was a little bit of naivety earlier in the market, and now that's kind of been replaced by a more sober-minded approach to say this.
• 13:00
  SPEAKER_02: But tell me more about what people's expectations are, and then um what you have to do to kind of uh mitigate or deal with those or manage those expectations.
• 13:10
  SPEAKER_01: Yeah, so so so of course, like expectations are high, really high, because like the uh information that we hear about Genai are so promising that that expectations are at so high high level, it's really hard to manage it.
• 13:25
  SPEAKER_01: So basically, our approach, because we we never know if we the Gen AI is really brand new technology, and we and we are not sure if it will work, if it works correctly with your data.
• 13:39
  SPEAKER_01: We need to check it.
• 13:40
  SPEAKER_01: So we every time we we we start a project with Gen AI, we we propose to do POC first.
• 13:47
  SPEAKER_01: So quick low budget uh uh project that will just show what kind of possibilities we have with Genai.
• 13:54
  SPEAKER_01: So, for example, if if you would like to develop rack system for your data, I would propose you to do it in like one or two months.
• 14:02
  SPEAKER_01: Just deploy the like default version of the system, fine-tune it a little bit, and just to check how accurate it is, just to give you to play with it, to check if this is enough for you.
• 14:12
  SPEAKER_01: And after that, we will create a report, what kind of techniques we we used, and what are our propositions to next, what are our next steps that we should take, and um and what could be expected accuracy if we put more effort to it, right?
• 14:28
  SPEAKER_01: Because if you would like to have like RAG system that is really accurate and you have complex data, it could take one year of like two or three engineers like doing fine-tuning, researching, and and making system accurate, right?
• 14:41
  SPEAKER_02: So, talk a little bit about um how rag is evolved.
• 14:46
  SPEAKER_02: So originally we were saying, well, ask your LLM for the answer, and the LLM would come up with an answer that may not be based on reality, right?
• 14:55
  SPEAKER_02: So it would hallucinate.
• 14:57
  SPEAKER_02: And then people said, Well, use a knowledge base.
• 15:00
  SPEAKER_02: And and then that kind of in the but then that fell back on the challenges of any basic knowledge base.
• 15:06
  SPEAKER_02: And what I saw initially were people just using full text search, keyword search, and that's gonna be as good as your keyword search is today, right?
• 15:15
  SPEAKER_02: If you don't have finely tuned content and you don't have things like a thesaurus structures, or you don't have semantic search, or you're not using parametric search or whatever, it's gonna be bad, right?
• 15:26
  SPEAKER_02: If if you have not, if you have crappy content and and you're using sparse keywords and ambiguous queries, it's gonna be like search, it's gonna suck like search sucks.
• 15:36
  SPEAKER_02: And then we got into advanced rag, which was which uh so talk about naive rag, advanced rag, and then talk a little bit about what modular rag is from your perspective.
• 15:47
  SPEAKER_02: So naive rag is just let's just go off and search.
• 15:51
  SPEAKER_02: We'll use the LLM to interpret the question, then we'll go to a data source, and then we'll retrieve that, and then we'll make a conversational with the LLM, right?
• 15:58
  SPEAKER_02: And then, but that had lots of drawbacks.
• 16:00
  SPEAKER_01: So, what was the next iteration or the next evolution in terms of yeah, so so so, of course, like the more advanced rag contains like things like re-ranker, like also the system that actually takes so model that actually takes the query, takes the retrieve chunks of data, like I don't know, 10 most probable or something like that, and then compare them and and choose the the answer.
• 16:26
  SPEAKER_01: This is something that can can increase accuracy.
• 16:29
  SPEAKER_01: You can retrain uh and you should retrain embedder to to fit your data.
• 16:33
  SPEAKER_01: This is also something that the advanced rack can can have.
• 16:37
  SPEAKER_02: Tell me more about that.
• 16:38
  SPEAKER_01: What's the so business?
• 16:40
  SPEAKER_01: Yeah, this is like currently we have really nice embedders, so simply models that that changes chunks of data, like text data, to just embedding, right?
• 16:50
  SPEAKER_02: Uh and then to a mathematical representation, yeah because all LLMs use uh operate, they're mathematical operators essentially, right?
• 17:00
  SPEAKER_02: They operate on numbers rather than words, and the words are converted to numbers.
• 17:04
  SPEAKER_02: So we convert our text to a vector, right?
• 17:08
  SPEAKER_02: And dimensional vector, right?
• 17:09
  SPEAKER_02: Lots of different characteristics, and then we bring that into the vector space.
• 17:13
  SPEAKER_02: That is called an embedding, right?
• 17:16
  SPEAKER_01: Yeah, yeah, exactly.
• 17:17
  SPEAKER_01: So so initially we just used like the the best embedder that were on the market, but like I don't know, one year ago the embedders were not really good.
• 17:28
  SPEAKER_01: So so every time you you you wanted to build a like high highly accurate rate rack system, you you needed to retrain this embedder with your data so simply.
• 17:38
  SPEAKER_01: So in a way, it will put embeddings for from question and proper answer close to each other and wrong answer far from each other, right?
• 17:47
  SPEAKER_01: So so so this is this is the the goal of of this this embedder.
• 17:51
  SPEAKER_01: Currently, we have like new embedders like e5 from Google, which is pretty nice, but still you can gain a lot of accuracy if you retrain the system with your data, right?
• 18:01
  SPEAKER_02: When you say train it with the data, what we've done is we've ingested the content along with metadata, so had a knowledge architecture that provided additional tags or additional features essentially of the content for an enriched embedding.
• 18:20
  SPEAKER_02: So when you're bringing it in, this is why knowledge architecture is important, right?
• 18:23
  SPEAKER_02: This is why knowledge bases are important and metadata is important.
• 18:26
  SPEAKER_02: If you have these things tagged with the right product model, right, and the right troubleshooting step and the right, you know, whatever the the uh uh error code it is, right?
• 18:36
  SPEAKER_02: Like I use that specificity to say an LLM will not know those things unless you tell it.
• 18:42
  SPEAKER_02: And the way you tell it is by tagging the chunks of content before you ingest it.
• 18:46
  SPEAKER_02: And then when you ingest it, you're bringing in the additional signals from that knowledge architecture to improve the ability for the LLM to make a distinction between a good and a bad answer.
• 18:59
  SPEAKER_02: Does that make sense?
• 19:01
  SPEAKER_01: Yes, of course.
• 19:02
  SPEAKER_01: This is this is one of the techniques that you can use to increase the accuracy.
• 19:05
  SPEAKER_01: And this is also why uh you still need like good engineers who are aware of different techniques and and can analyze your data, check which technique could could be applied here, uh, because not it is not always the case that you can you can create some metadata or you can filter out some some data during retrieval process.
• 19:24
  SPEAKER_01: Sometimes it is possible, sometimes not.
• 19:26
  SPEAKER_01: You you you you always need like good people who who who knows where to use which techniques, right?
• 19:32
  SPEAKER_01: So yeah, but but yeah, this is this is this is one of the techniques that can increase accuracy of the embedder system.
• 19:37
  unknown: Yeah.
• 19:37
  SPEAKER_03: You started to pull on a thread earlier around asking about the models.
• 19:43
  SPEAKER_03: And are you taking a foundation model and and doing your fine-tuning against that?
• 19:47
  SPEAKER_03: Are you creating your own model?
• 19:48
  SPEAKER_03: Are you doing continuously fine-tuning on top of those models?
• 19:52
  SPEAKER_03: Because it seems to me to Bartek that there's this super art form or combination of what you're talking about, where you have potentially human in the loop, right?
• 20:02
  SPEAKER_03: Or the validation of the data in the beginning, and then you have this notion of just the the ability to figure out what to do with the model itself and what model is going to work the best and all that kind of stuff.
• 20:13
  SPEAKER_03: And then also then the fine-tuning and/or the continuous fine-tuning that goes into it.
• 20:18
  SPEAKER_03: And do you need to add in human in the loop, right?
• 20:20
  SPEAKER_03: Because I think what what we're also talking about is it depends on the use case as well, on how those combinations need to come together.
• 20:28
  SPEAKER_03: Will you talk a little bit about that?
• 20:29
  SPEAKER_03: Because I'm assuming you have some models in production and going from POC to production, you've had to learn a lot of these different types of techniques that need to be applied.
• 20:40
  SPEAKER_03: So we, I don't know if you have some use cases in mind as examples, but what does that look like in terms of pulling that art form together?
• 20:47
  SPEAKER_01: Yeah, but so like you said, like every use case is different and you need to apply different techniques.
• 20:53
  SPEAKER_01: For example, we we we um currently for one for one of our clients, we uh work with probably the most advanced search engine and podcast environment.
• 21:04
  SPEAKER_01: Um so for a company called Podemo, we we developed like the search engine.
• 21:09
  SPEAKER_01: And and the the one thing that we uh we needed to do there is along with the classic rack system, we needed to create fusion with keyword search, I would say, right?
• 21:18
  SPEAKER_01: And the second thing we needed to also take into consideration is the way people usually search podcasts.
• 21:24
  SPEAKER_01: So they they they they usually do it in a Google-like style, I would say.
• 21:28
  SPEAKER_01: So they, for example, write half of the word because Google will know the answer anyway, right?
• 21:34
  SPEAKER_01: So also you need to cover things like that.
• 21:36
  SPEAKER_01: So depends on the use case, you need to to to to use different techniques.
• 21:40
  SPEAKER_01: But more important is that you need to have these users in the loop, right?
• 21:44
  SPEAKER_01: So you need to check what they actually uh write to your system in what way, what they are looking for, and you need to adjust your system to to your users, right?
• 21:55
  SPEAKER_01: Because, like for RAC system to to be successful.
• 22:00
  SPEAKER_01: You need three very important things.
• 22:03
  SPEAKER_01: One is the accuracy, of course.
• 22:04
  SPEAKER_01: Second is length lengthening latency, because you can easily create an accurate system that will be really, really slow.
• 22:12
  SPEAKER_01: Uh, you can put a lot of different models, double checks everything and stuff like that, but it you you will wait 15 seconds for the answer, right?
• 22:18
  SPEAKER_01: And it will make no sense.
• 22:20
  SPEAKER_01: And the second thing is really nice user interface.
• 22:22
  SPEAKER_01: These three things is extremely important to make system user-friendly and actually to to to make people use it.
• 22:31
  SPEAKER_01: Because without it, even if you have a highly accurate system, they they they won't like to use it.
• 22:35
  SPEAKER_01: And actually, people are the biggest barrier right now, right?
• 22:38
  SPEAKER_01: Because they sometimes they of course there are people that want to use the most advanced technology and they will use everything even if there is no sense of it.
• 22:47
  SPEAKER_01: But um there's a there's a big uh group of people that simply don't like changes, right?
• 22:54
  SPEAKER_01: And and to convince them that uh that they should use uh AI technology, you need to create a system that is accurate, super fast, and really nice in terms of interface, right?
• 23:05
  SPEAKER_03: I love that I love that we have these like 70 billion parameter models, these 250 billion parameter models and stuff.
• 23:11
  SPEAKER_03: And to your point on latency, I had a customer I was working with that decided to throw their entire product catalog, e-commerce site, and throw their entire product catalog at the LLM because they might have a search about one product, super inefficient.
• 23:27
  SPEAKER_03: But because the parameter window was that big, then they could do it.
• 23:32
  SPEAKER_03: But you know, this is what kind of an add-on to that question is so also how do you balance cost optimization there?
• 23:40
  SPEAKER_03: Because part of that is I can slam and cram this against these large models all day long, but it's so expensive as well.
• 23:47
  SPEAKER_03: And I would say, you know, for me, that is that add-on to the user experience is how do you also make it so that it's optimized so that you can utilize it for billions of people or whatever the use case is and not kill yourself.
• 24:02
  SPEAKER_01: Yeah, so so of course, like quantization is is one of the strategies that makes system extremely fast, and you can you can actually apply double checks and and and and three rankers and and other techniques only if you if you are able to quantize your model and and you still can achieve good results because without this system will be too slow or it will not fit to your memory, or you would need to apply like a huge machine that that that that has a lot of memory to handle your your system, right?
• 24:32
  SPEAKER_01: And if you would like to make your system scalable, serve to a million users, you definitely need to do quantization of your model, yeah, and and queries and everything, yeah.
• 24:40
  SPEAKER_01: And unfortunately, these models are very sparse, so the embeddings are very sparse.
• 24:46
  SPEAKER_01: So quantization doesn't harm it too much, yeah.
• 24:49
  SPEAKER_03: Yeah, that's good.
• 24:50
  SPEAKER_02: So um we talked about naive rag, we talked about advanced rag.
• 24:55
  SPEAKER_02: Uh, do you want to talk a little bit about modular rag and your opinion of how that works and and how effective it is, and and so on?
• 25:05
  SPEAKER_01: Yeah, so so so what this is this is definite yeah, so yeah, the definition.
• 25:10
  SPEAKER_01: So I think that this is this is the most unmature type of rag that especially in terms of like a production use case where we can see how it works in in production.
• 25:19
  SPEAKER_01: But yeah, this system contains simply a lot of modules.
• 25:22
  SPEAKER_01: For example, it not only looks for the answer in your knowledge base, it can search in the in the internet, it can combine those two sources, it can you can add other sources, uh apply uh free rankers, like yeah, you you you you do fusion from different types of racks actually.
• 25:40
  SPEAKER_01: And um and and uh and uh and the issue with it is that it all makes it slow, right?
• 25:46
  SPEAKER_01: So so so there is not much use cases, I would say, that uh you can sacrifice latency for a higher accuracy.
• 25:55
  SPEAKER_01: Uh of course there are use cases like that, but um I would say that 95% of our projects are rather system that needs to work really fast, right?
• 26:08
  SPEAKER_01: So so there's no way of doing modular rag uh in that case.
• 26:12
  SPEAKER_02: So so we've been using it a little differently.
• 26:15
  SPEAKER_02: We've used it as kind of a template using templated prompts and programmatic prompts that are cycling through data and using modular rag to ingest lots of different data sources, so it becomes more of a batch processing uh tool rather than a real-time retrieval tool.
• 26:33
  SPEAKER_02: And what that is using doing is it's it's using it to improve the quality of data.
• 26:38
  SPEAKER_02: So we we have one use case where we had very, very sparse data on the product record, and then we enriched it with FCO terminology, with a richer product description, with additional attributes, categories, keywords, and then made that audience specific.
• 26:54
  SPEAKER_02: So it was a good use case, but again, it's a different use case because it's not it's talking about using it not in a real-time fashion, but more of a batch processing fashion.
• 27:04
  SPEAKER_02: And also from those agent approaches, too, right?
• 27:07
  SPEAKER_02: Because you're using to often do a particular process.
• 27:11
  SPEAKER_01: Yeah, yeah.
• 27:12
  SPEAKER_01: Like the the the issue you describe, well, like the the data describes seems very complex.
• 27:17
  SPEAKER_01: So so it's probably a good use case for modern rack to to to to to use to to solve this issue, right?
• 27:24
  SPEAKER_02: So tell us a little bit more about you, what you do for what do you do for fun, and uh tell us a little more about um your journey of how you how you got to what you're doing.
• 27:34
  SPEAKER_03: Uh who is bar tech?
• 27:35
  SPEAKER_02: Right, sorry.
• 27:37
  SPEAKER_03: I said, yeah, who is bar tech outside of AI?
• 27:40
  SPEAKER_03: And maybe you're an outside of AI.
• 27:42
  SPEAKER_01: Um I don't think I have there's any bar tech outside of AI.
• 27:48
  SPEAKER_01: I would say that my in my like my hobby is like like like I said, I I like to write models on stock exchange, right?
• 27:55
  SPEAKER_01: So to play on the market and and stuff like that.
• 27:57
  SPEAKER_01: So this is actually my passion uh for a very long time, and I'm still writing my own models, applying new techniques.
• 28:04
  SPEAKER_01: This is also actually the way I'm staying up to date with the current technology, right?
• 28:08
  SPEAKER_01: And because now in my role I'm far from the coding and I'm more close to the business, but I really like the technology and I need to be close to it to be effective in my current role.
• 28:20
  SPEAKER_01: So so this is also my playground where I can apply, I don't know, new new reinforcement learning techniques and and and stuff like that.
• 28:26
  SPEAKER_01: Yeah.
• 28:27
  SPEAKER_03: So so what do you think of then all these AI bots that are out there?
• 28:31
  SPEAKER_03: I mean, obviously most of them are going to be pattern recognition, but I love it that you see a lot of rhetoric and a lot of folks that are that are out there.
• 28:38
  SPEAKER_03: Hey, look at my AI bot.
• 28:39
  SPEAKER_03: It's made me so much, you know, thousands dollars per month on these bots and everything else.
• 28:43
  SPEAKER_03: And I can only imagine that it's literally looking at the patterns of the candlesticks and everything else and just trying to see when leading indicators, lagging indicators, all that kind of stuff.
• 28:53
  SPEAKER_03: I don't know what your thoughts are there.
• 28:55
  SPEAKER_03: It's just it's it's one of those things that I watch too, and I'm like, hmm, a little suspect, but at the same time, I think you have to build it yourself.
• 29:01
  SPEAKER_03: Go ahead.
• 29:02
  SPEAKER_01: Yeah, so I'm in this like uh industry for a very long time, and I don't believe in any kind of statement like that.
• 29:09
  SPEAKER_01: I know a lot of people who who who created like I wouldn't say bot like that, but but strategies that works, then they don't tell anybody about it, right?
• 29:18
  SPEAKER_04: Yeah, of course.
• 29:21
  SPEAKER_01: Yeah, the only way you would like to tell more people that that you have successful strategies because you want to take their money and invest their money, right?
• 29:30
  SPEAKER_01: But this is for uh like uh I would say big hedge funds and and stuff like that.
• 29:34
  SPEAKER_01: So this is of course obvious.
• 29:36
  SPEAKER_01: But if we're talking about like uh proprietary trading and day trading and stuff like that, when you just like have your own strategy for yourself, you don't share your strategy.
• 29:44
  SPEAKER_01: Simply you don't share your results.
• 29:46
  SPEAKER_01: I don't think I I know probably maybe I'm not up to date with the X platform and and and the and the you know people who and their claims there.
• 29:55
  SPEAKER_01: Maybe there are some you know superstars that that just like to be famous.
• 29:59
  SPEAKER_01: I don't know.
• 30:00
  SPEAKER_01: But my my experience is that people who who knows how to earn money on the market stay silent.
• 30:06
  SPEAKER_03: Well, it's fascinating.
• 30:07
  SPEAKER_03: So thanks for your answer.
• 30:09
  SPEAKER_02: Quick question.
• 30:10
  SPEAKER_02: So sometimes I like to ask this if you uh were able to go back to yourself and when you graduated college, uh is there any advice you would uh you would provide yourself looking back into your life?
• 30:23
  SPEAKER_01: Yeah, I don't know.
• 30:24
  SPEAKER_01: Um it's it's it's a hard question.
• 30:26
  SPEAKER_01: I I did it a long time ago.
• 30:28
  SPEAKER_01: I don't know if I think the current world is different, right?
• 30:30
  SPEAKER_01: But yeah, probably the the thing that is important is that I've I'm working now in the industry that is totally different than my educational background, right?
• 30:40
  SPEAKER_01: And and this is something that probably will happen to a lot of folks that currently graduate, right?
• 30:45
  SPEAKER_01: Because AI will like change the way we work simply.
• 30:49
  SPEAKER_01: So what I would uh recommend to people is that don't hesitate to to change your background.
• 30:56
  SPEAKER_01: If if if you see some potential somewhere somewhere else, like go for it, right?
• 31:01
  SPEAKER_01: This is one thing.
• 31:01
  SPEAKER_01: And second thing, if if you if you feel that you can do something good, um also go for it, even though you are getting reviews uh um many times, right?
• 31:12
  SPEAKER_01: Before I started my career and and and and then stock exchange, I I did a lot of strange things uh because uh uh I it happens to me that I entered the the the labor market during the financial crisis uh 2007.
• 31:29
  SPEAKER_01: So so it was really hard to get a job in finance that day.
• 31:34
  SPEAKER_01: Um, so so so yeah, I I needed to do a lot of like uh really, really simple jobs.
• 31:39
  SPEAKER_01: Uh and and actually I I could stay there because I got better and better at that.
• 31:44
  SPEAKER_01: But I know but but I but I had just like different dreams, so I decided to pursue it, and I would recommend anyone to pursue it.
• 31:51
  SPEAKER_02: Yeah, what's the worst job you ever had?
• 31:54
  SPEAKER_02: The worst job I ever had, yeah.
• 31:57
  SPEAKER_02: Oh my god, yeah.
• 31:58
  SPEAKER_02: I'm well you're coming in the muffin.
• 32:01
  SPEAKER_02: I cleaned a muffin factory.
• 32:03
  SPEAKER_03: Uh I shoveled horseshit.
• 32:06
  SPEAKER_01: Keep going.
• 32:07
  SPEAKER_01: Yeah, okay.
• 32:08
  SPEAKER_01: So so I think I was at the worst job I had was I was working in a huge like um uh mail center, and for the uh whole night I was like packing the big cars with the packages, right?
• 32:26
  SPEAKER_01: For the 12 hours, right?
• 32:27
  SPEAKER_01: Uh I remember this was probably my first job up there.
• 32:30
  SPEAKER_01: I graduate the college.
• 32:31
  SPEAKER_01: Kind of like working for Amazon today, right?
• 32:34
  SPEAKER_01: Yeah, exactly.
• 32:35
  SPEAKER_01: So it was yeah, it was exactly like that.
• 32:38
  SPEAKER_01: Um, yeah, and it was very hard.
• 32:40
  SPEAKER_01: Uh it was very hard, yeah.
• 32:42
  SPEAKER_02: So let's see where people can find you.
• 32:45
  SPEAKER_02: You can uh go to stxnext.com.
• 32:49
  SPEAKER_02: That's the company, and then uh you are on LinkedIn, and it's just B-A-R-T-E-K-U Rosak uh R O S Z A K.
• 33:00
  SPEAKER_02: That will all be in the show notes.
• 33:02
  SPEAKER_02: And Partek, I want to really thank you for your time today and for sharing your expertise.
• 33:06
  SPEAKER_01: Yes.
• 33:08
  SPEAKER_01: Yeah, thank you very much.
• 33:09
  SPEAKER_01: Thank you for having me, and it was a pleasure.
• 33:11
  SPEAKER_02: It's been great.
• 33:12
  SPEAKER_02: And uh thanks every to everyone who's listening, and uh thank you, Carolyn and Chris.
• 33:16
  SPEAKER_02: Uh and again, Partek, it was uh great to have you and uh look forward to continuing our conversations.
• 33:22
  SPEAKER_02: Thank you.
• 33:23
  SPEAKER_02: Thank you very much.
• 33:24
  SPEAKER_00: Thanks, Partek.
• 33:26
  SPEAKER_00: Visit early.com to find links to the full podcast on all audio platforms to listen on the go.
• 33:32
  SPEAKER_00: Thank you for watching.
• 33:34
  SPEAKER_00: Contact us today at infoearly.com.