Latest / Agentic Conversations (formally mlops.community) / Experiment Tracking in the Age of LLMs // Piotr Niedźwiedź // #168

Transcript

Experiment Tracking in the Age of LLMs // Piotr Niedźwiedź // #168 · Agentic Conversations (formally mlops.community)

• 0:00
  Hello and welcome everyone. This is the ML OPS Community
• 0:03
  Podcast. I'm your host Dimitrios and
• 0:05
  today I was joined by Vishnu as a Co host but he had to bounce
• 0:09
  real fast. So I am doing this one solo.
• 0:11
  The recap the intro of our conversation with the CEO of
• 0:16
  Neptune Peter. I found a few things fascinating
• 0:19
  about this conversation. I think Vishnu had some key
• 0:23
  takeaways that he relayed to me to.
• 0:25
  The first thing that I loved is how he is navigating and how he
• 0:30
  is effectively steering the ship through the incredible amount of
• 0:36
  changes that are happening when it comes to the machine learning
• 0:41
  and AI space and where he plays and what his tool has
• 0:45
  traditionally played in this idea of experiment tracking or
• 0:50
  metadata management. And so being a CEO of a company
• 0:54
  that is trying to navigate these waters, I thought he had some
• 0:58
  gems that he took away or that I took away from it.
• 1:01
  And I don't want to give any spoilers, so I'm going to let
• 1:03
  you come to your own conclusions.
• 1:05
  But it's nice to see how he thinks and see how he decides to
• 1:09
  make decisions. That being said, of course we
• 1:14
  talked about traditional ML OPS and Neptune and the way that
• 1:19
  they have brought themselves to the table and now what they're
• 1:23
  looking at when it comes to LMS and how that changes in the
• 1:27
  product vision and in the product that they are creating
• 1:30
  for data scientists and machine learning engineers.
• 1:34
  So let's jump into this conversation before we do.
• 1:36
  I have 1:00 ask for you just one, and that is you share this
• 1:40
  episode with just one friend so that we can keep the good words
• 1:45
  spreading about all of this knowledge and wisdom we are
• 1:49
  sharing on this podcast. That's it.
• 1:51
  Let's great to have you here. I'm super excited to talk all
• 2:01
  about LMS and how you all had Neptune are doing some wild
• 2:05
  stuff with that and where your vision is.
• 2:07
  And you just said something before we started talking that I
• 2:10
  feel like we have to address right now.
• 2:12
  It's almost the elephant in the room and you studied with Sam
• 2:16
  Altman and. Can you not with Sam?
• 2:19
  We vote. That's a little different there.
• 2:23
  OK, Who's voice tick and what is going on you connected with?
• 2:27
  Sam, OK. OK, Voytek is one of the
• 2:30
  cofounders of Open AI and. He he's more like he's, I think
• 2:37
  he's still leading the research group there.
• 2:40
  And so I would say initial brain behind when it comes to research
• 2:44
  and and science. Actually there are quite a few
• 2:50
  of my friends from studies and also from.
• 2:52
  Competitive programming times because I used to be.
• 2:56
  Ohh, let's call it a professional player in
• 3:00
  competitive programming. It is.
• 3:02
  It was not about data science. It was not covered.
• 3:04
  It was before it was classical algorithms.
• 3:07
  By the way, there is there is my professor from from from when
• 3:12
  from times that I studied math and computer science he.
• 3:15
  He had a kind of upgraded version of a Turing test.
• 3:21
  And the upgraded version is when like the the bolt, when when the
• 3:26
  algorithm would be able to qualify to national level of
• 3:31
  Olympian informatics. And we are not there yet, like
• 3:34
  this question was asked today to Voytek, we are not there yet.
• 3:38
  Voytek predicts that. In this decade it will happen
• 3:42
  and but it is not yet obvious how it will be done so even for
• 3:47
  you guys who are creating. Core behind large language
• 3:51
  models. The capabilities and the
• 3:55
  projection like we definitely people feel that.
• 3:59
  We are not. It is not the end.
• 4:01
  We will be able to do more. Like he invitingly the Today
• 4:05
  and. Comparison to sigmoid function
• 4:08
  and where are we in the curve of progress?
• 4:11
  If it was represented by sigmoid function, he bet that in the
• 4:15
  middle. So it means that quite likely
• 4:17
  we'll see significant progress in the next few months, years,
• 4:23
  but it should flatten. I think it is similar how to
• 4:28
  deep learning wave maybe seven years ago or so.
• 4:31
  When deep learning was kind of in the basement for years, like,
• 4:36
  yeah, the techniques, your networks are all right, but
• 4:38
  nobody was talking about it. Then there was like, we notice,
• 4:44
  OK, this is possible, this is possible.
• 4:46
  More people will focus on it. There were more, more
• 4:48
  investment. And then the progress started
• 4:50
  like booming. And to some extent like.
• 4:54
  Yes, we can always say that large language models are still
• 4:58
  deep learning models. But this is a different game.
• 5:00
  It's a different scale. Yeah, I think it is not fair to
• 5:03
  really compare them, even though they are.
• 5:06
  Technically wise, some extent similar, yeah.
• 5:10
  So you you undersell yourself a little bit there on the
• 5:14
  professional aspect of your informatics abilities.
• 5:19
  I read somewhere that you were what you were an Olympiad for
• 5:24
  Poland in your informatics. What's that?
• 5:28
  All that I I had silver medal on the national level while I was
• 5:33
  in high school. It is weird by the way.
• 5:36
  They like Polish, independent formatics and and this whole
• 5:39
  competitive programming. Community it is what?
• 5:44
  It gave me so much because on one hand.
• 5:48
  I because of that I was able to go to any university in Poland,
• 5:52
  so I picked the best one for for this type of studies.
• 5:56
  And so consequently I was able to meet.
• 6:00
  Like the smartest guys in in in this field national and but also
• 6:05
  internationally because when I was a student I was also
• 6:08
  competing in competition like Google Cogent Facebook Hacker
• 6:11
  Cup. So it gave me, I would say.
• 6:16
  Some level of confidence. That's yeah I I am also quite
• 6:21
  smart right. So I can try I I'm, I'm not
• 6:24
  afraid to to try to build my own company etcetera, etcetera so.
• 6:29
  It gave me a lot. Talk to us a little bit more
• 6:32
  about building your own company. You've done a few companies now.
• 6:36
  Yeah, exactly. You've got a few.
• 6:38
  Tell us about those. And then now you're on Neptune
• 6:41
  and how did that transpire? So I started my, let's say,
• 6:46
  entrepreneurial background. And what I was too studying one
• 6:51
  day I decided that I am done with competitive programming
• 6:55
  because. Quite likely I after being top
• 7:00
  30 worldwide and Google Code Jam I.
• 7:03
  I got offer from yeah, most of the.
• 7:07
  Big companies in the Valley. So this kind of goal being able
• 7:11
  to work as a software engineer in companies that I respect and.
• 7:17
  Was kind of done so. So I decided let's start my
• 7:20
  first company at that time. I was still student.
• 7:23
  I wanted to finish my studies so.
• 7:26
  And I studied in Poland, and in Poland, the VC ecosystem at the
• 7:29
  time didn't really exist. Maybe maybe there was something
• 7:32
  that? So I I I had to have money to to
• 7:38
  build stuff and. So I decided to create the first
• 7:43
  company that will be able to generate money.
• 7:47
  To build. Brother company and and it is
• 7:50
  how my first company like I'm co-founder codeline and this
• 7:54
  company that specialised in. Software defined networking.
• 7:59
  So networking, but not like not hardware, more software.
• 8:04
  Something in between hardware and software.
• 8:07
  And it was 12 years ago in the like it's supposed to be a small
• 8:11
  company. At the beginning, it was the
• 8:12
  plan. But today it is over 400 people
• 8:16
  company, highly profitable while growing.
• 8:19
  I'm not obviously anymore like operationally involved because
• 8:25
  I'm focused on Neptune, but my brother is CEO and.
• 8:29
  My friends who who Co founded this company with me are like we
• 8:33
  are still close and I still major shareholder.
• 8:35
  So so it was one company a few years later I think 4 like so
• 8:40
  8-7 years ago. I Co founded Deep Sense AI that
• 8:45
  would be one of the leading consulting firms in Europe.
• 8:49
  The idea like let's focus on data science.
• 8:53
  Actually, I need to give credit to Vortex that Emma again.
• 8:58
  It was after we met in San Francisco, and he was at the
• 9:00
  time. Pursuing PhD and around deep
• 9:04
  learning. And and he was always I I need
• 9:08
  to admit that Voytek was always. A believer at Visioner when it
• 9:14
  comes to capabilities of those models today, it is easy to be
• 9:18
  to say yeah, like they are amazing.
• 9:20
  Whatever is possible, like it is easy.
• 9:22
  It was. But eight years ago, it was not
• 9:25
  so easy. And and yeah and and I kind of
• 9:30
  felt that he might be right. And when I look at the at the
• 9:33
  time, cybersecurity was big. I think blockchain maybe it was
• 9:37
  before blockchain wave. Definitely big data or
• 9:40
  something, but. I felt that being able to do
• 9:45
  something like relevant with the data.
• 9:48
  Quite likely the game will be a game changer, so.
• 9:52
  So away how you would like. If I were just an individual
• 9:55
  person with the company, quite likely I would start studying
• 9:58
  deeper deep learning as a as a as a founder with that has that
• 10:03
  had already a company. I decided to do both on one
• 10:06
  hand. I wanted to get hands on and
• 10:09
  understand this concept deeper. On the other hand, I thought
• 10:13
  maybe it would be better to do it together with other folks.
• 10:16
  So I cofounded a company that's decided to specialise just in
• 10:20
  this field and we started from participating in Kaggle
• 10:24
  competitions to get to the level that we say, OK, we can build
• 10:29
  world class models, maybe it's time to start providing service
• 10:32
  around it. So it is how it is.
• 10:35
  The other company and from deep says it is the place where
• 10:39
  Neptune was basically created, the first version.
• 10:42
  It was internal. LOL it wasn't, it wasn't my idea
• 10:44
  to be honest. One of the one of the data
• 10:48
  scientists who also participated with me in Polish and
• 10:50
  informatics. He created this experiment
• 10:53
  tracker combined with with some infrastructure management layer
• 10:58
  and so because because yeah, today some.
• 11:02
  It would be rather a bad idea because cloud providers.
• 11:05
  Made the API and the way you can use hardware for training way
• 11:09
  easier. But seven or six years ago it
• 11:12
  was. It was not like that.
• 11:15
  It was very engineering heavy type of API and experience so so
• 11:19
  yeah it is how it started. I can tell you more about it,
• 11:22
  but Long story short from 2 consulting companies.
• 11:26
  I finally got like wafi solved the problem and felt that this
• 11:32
  problem can be something that is not just tips insulated.
• 11:36
  I talk with my friends who are data scientists or data science
• 11:39
  leader in other companies. How do you like?
• 11:42
  Yes, you're building models, how do you manage it?
• 11:45
  And I've heard very often what is still I would say pretty
• 11:49
  popular solution. Today I'm using a spreadsheet
• 11:53
  like homegrown type of solutions.
• 11:56
  So it was. That is usually a good signal,
• 12:00
  but there is that if you hear it.
• 12:03
  That many people do. It's that signal for a problem,
• 12:07
  a different struggle or there would be a sustainable business
• 12:10
  around it, right? But for a problem, it was a good
• 12:12
  signal. So it is how Neptune was
• 12:15
  started. So talk to me real fast about
• 12:18
  this. It's a bit of a tangent, but I
• 12:19
  got to ask it. When you meet somebody outside
• 12:23
  of Poland, yeah, like in the US and you are chatting with them
• 12:27
  and they also have passions that line up with yours.
• 12:30
  Is it like you all are in this small club?
• 12:32
  It's like we're the two Polish people that are in the US that
• 12:36
  actually care about deep learning and you give each other
• 12:38
  a bit of a head nod and you're on the same team.
• 12:42
  To be honest, it wouldn't be. I I think it is it is changing
• 12:47
  now because. It was never a Polish thing for
• 12:51
  me, so. I didn't care so much, right?
• 12:55
  Like if the guy was from Poland or not and but.
• 13:00
  It used to be more if I met somebody who was really doing
• 13:04
  something with machine learning. You're even a year ago in a kind
• 13:09
  of random occasion and not talking conference around
• 13:13
  develops, right? But random occasion like it was
• 13:17
  still a small clock of people who were following this space.
• 13:21
  We've large language model. It has changed like.
• 13:24
  Two days ago I was using Uber. And the driver was.
• 13:29
  Brought the topic of of AI, so it is a signal that that.
• 13:35
  It got quite popular. Yeah, it's gone mainstream.
• 13:38
  And yeah, we're not cool anymore because it's too popular.
• 13:42
  We're not the. People Yeah, well dude, there is
• 13:45
  something that I want to mention before we jump into this and
• 13:50
  that I would love to get your thoughts on.
• 13:51
  It's around the idea of how you've adapted with Neptune,
• 13:55
  because you mentioned before that the original product was
• 14:00
  built for a pen that now the cloud providers abstract away.
• 14:05
  Yeah. How have you looked at the
• 14:07
  adaptability of the product that you're building?
• 14:11
  So I think it is is pretty challenging in this space.
• 14:16
  To run product. Because and and I think it is.
• 14:22
  There is a new wave of change of changes.
• 14:26
  And those kind of observations can be still relevant so.
• 14:30
  I think that very important that it is on one hand, very easy
• 14:35
  today and it used to be and it was easy five years ago.
• 14:40
  To see problems. When I like in my team, I'm
• 14:45
  trying to explain people who reports to me that.
• 14:49
  Seeing problems in a startup. Is not a unique skill.
• 14:56
  So I would say the same here, like seeing problems in the
• 14:59
  space of myopes or large language model OPS if we like,
• 15:03
  depends how we would define it. Is not a is not a challenge.
• 15:08
  The challenging thing is to decide.
• 15:12
  What problems are temporary problems?
• 15:16
  Yes, I can solve it right. I'm quite rather if I am
• 15:20
  students and I am still kind of skillful in coding.
• 15:24
  Quite likely I would be able to solve, not in a super generic
• 15:28
  way, particular hack, particular problem quickly, faster than a
• 15:32
  cloud provider, right? But.
• 15:36
  We'll crawl cloud provider for instance.
• 15:38
  Will it dissolve in heavier in a year?
• 15:43
  What is my defensibility so? Yes, you definitely like one
• 15:49
  thing. You need to look at the
• 15:49
  problems, but it's not super challenging.
• 15:53
  You need to be able to figure out what is the solution.
• 15:55
  Maybe a little bit harder, but still, I won't say this rocket
• 16:00
  science very often. But will you be able to?
• 16:06
  Like build a sustainable business around it so you will
• 16:10
  be able to make iterate making better, like really do something
• 16:15
  in the long run with that. Is a different story, so.
• 16:20
  And it is challenging. Maybe for some people even this
• 16:23
  is not challenging, but I am at the level where this is where I
• 16:26
  feel challenged. And and I think a framework to
• 16:30
  think about it. That I realise relatively late.
• 16:35
  Is a way like it is I I think it is.
• 16:38
  One-on-one of product management, but.
• 16:42
  But I don't have product management background, I have
• 16:44
  engineering background. So understanding what are the
• 16:49
  like, what is the essence? What are the main jobs to be
• 16:53
  done? That we want to solve and why we
• 16:58
  should solve it. Why?
• 16:59
  Those jobs shouldn't be solved by somebody else sometimes.
• 17:04
  And if you look at the maybe landscape, competitor landscape
• 17:07
  from this perspective, you will realise that.
• 17:10
  You'll notice this always competing with let's say
• 17:12
  envelopes layers. I wrote a blog post about DeVos
• 17:17
  space and connection to envelope space that was.
• 17:23
  Not every folk in envelopes community recycle its received
• 17:27
  its positively because why? Why we are bringing DevOps?
• 17:32
  We are, we are we are envelopes engineers.
• 17:35
  Yes, it's true. It's specialisation.
• 17:37
  I'm saying that that. Having just a developed
• 17:41
  knowledge is enough, but. I think you need to look from
• 17:45
  the perspective of like you need to think about.
• 17:50
  About the core problems. And and I see a lot of a lot of
• 17:55
  similarities to DeVos when it comes to core of the problems.
• 17:58
  But yes we are when it comes to way how we are solving it, the
• 18:02
  there are differences in details.
• 18:05
  I see something to some extent similar with with large language
• 18:09
  models and how we are. Going to operate then like it is
• 18:14
  also it's another another another story.
• 18:16
  But when it comes to the product, going back to the
• 18:19
  original question, I think. I think it is essential to
• 18:24
  really understand what is the scope of jobs to be done.
• 18:28
  Your product is supposed to. Surf.
• 18:32
  And being super focused on that. Explicit about that.
• 18:37
  Also explicit when you communicate to the customers.
• 18:40
  In this space it was also easy. I remember it was very tempting.
• 18:43
  We had we we were able to close a bigger deals with enterprises,
• 18:49
  but they had needs. That I felt that they are not in
• 18:54
  line with the core jobs to be done of formation.
• 18:57
  So. We decided to pass on those
• 19:00
  deals even though closing one deal would double the revenue.
• 19:04
  So there was a pressure to do it right.
• 19:07
  But I think that's with envelopes and large language
• 19:12
  model OPS. It is a marathon.
• 19:14
  It is not. The game of 1 takes it all so.
• 19:18
  If you communicate to your shareholders, to your investors.
• 19:22
  What is the strategy you can you can make those decisions and and
• 19:26
  be careful what type of problems you're jumping to solve.
• 19:30
  It's very tempting, but very, very risky.
• 19:34
  I think that is a fantastic introduction and summary of
• 19:38
  product management in the context of machine learning and
• 19:40
  really like what you said about what are the temporary problems
• 19:43
  and what are the future problems.
• 19:44
  And it's clear that when you started this conversation by
• 19:47
  saying I wanted to start a product company but I didn't
• 19:49
  have the cash and you're clearly now trying to run a very product
• 19:52
  oriented company. You're thinking in a very
• 19:53
  product oriented way. And I hope the listeners
• 19:56
  appreciate it as much as I am because it's always fascinating
• 19:58
  to hear like real product visionaries think in terms of
• 20:01
  that I'm actually reading. Funnily enough, Steve Jobs is.
• 20:04
  Part biography right now at Walter Isaacson one, and what
• 20:07
  you're saying, it just meshes so well with the way that Steve
• 20:10
  thought about product, right? Because those those guys are
• 20:14
  like those, they are not but I say those guys and say Steve
• 20:17
  Jobs, OK there was one Steve job, but if you for instance
• 20:23
  like I really like. You can find out all over
• 20:26
  YouTube. Record like there was an
• 20:29
  interview. I think it was 1997 with Jeff
• 20:31
  Bezos. It builds, yeah.
• 20:34
  It bolts Internet and and they they I think they bought or
• 20:40
  built some centres brick and mortar right and.
• 20:46
  And Jeff was asked like, but you're supposed to be Internet
• 20:49
  company. What's What you're investors
• 20:52
  when you're building physical buildings?
• 20:55
  And he said something like Internet schmear Turnet.
• 20:59
  And what's important is what are like the core, core values we
• 21:04
  are providing to our customers and, he says, vast selection.
• 21:10
  Low price and and quick delivery and those are the fundamentally
• 21:16
  the most important three traits our company or features or
• 21:21
  company should provide to our customers.
• 21:23
  And I don't care whether it's via Internet or via building a
• 21:27
  building, right? And this this understanding of
• 21:30
  the like I would say noise cancellation or how we can,
• 21:35
  there's a lot of faults, a lot of ideas how we can really see
• 21:41
  what is the core and focusing just on that, it's simplify
• 21:45
  thinking so. I think that the leaders like
• 21:49
  Steve, like Jeff. They had this ability like they
• 21:55
  were really, really capable of. Understanding what is vital.
• 22:00
  Yeah. Yeah.
• 22:02
  And now I'm going to ask you the question, right.
• 22:05
  I'm going to ask you you the tough question here, which is
• 22:08
  the way that we build machine learning products is poised to
• 22:12
  change, right? That is what the promise of
• 22:15
  large language models are and that is what the promise of this
• 22:18
  new era of foundation models are.
• 22:20
  You don't need to build your own model from scratch necessarily.
• 22:23
  There are large models that truly can generalise in ways
• 22:26
  that are both exciting and almost scary.
• 22:30
  So for you as a company that exists to provide infrastructure
• 22:34
  and as you say, help data scientists, be data scientists,
• 22:38
  how has your product vision changed or how has it stayed the
• 22:42
  same in the context of these advances?
• 22:45
  I I look at that from this perspective.
• 22:47
  I start from maybe it sounds cliche, but if I think that we
• 22:53
  have pretty well thought. Compound emission.
• 22:57
  And this is something that you have mission.
• 23:00
  It should change sparingly, really sparingly.
• 23:03
  You should really be careful with that rollback vision or
• 23:07
  vision for the company still should be relatively stable, but
• 23:11
  you can adjust it a little bit more frequently.
• 23:13
  So when it comes to mission. Hmm, I don't see a huge change.
• 23:20
  And what is the mission then, right?
• 23:22
  Maybe I should see it. So we stated I wouldn't quote
• 23:26
  the 1:00 to 1:00, but basically. We want to give.
• 23:31
  MO or support machine learning teams.
• 23:34
  Maybe they should be called AI teams.
• 23:36
  If I'm talking about large ranking models, maybe machine
• 23:39
  learning is too narrow. But putting this aside.
• 23:42
  We want to provide them the same or help them to get the same
• 23:45
  level of control and confidence. When they are building.
• 23:49
  And testing. Their models like it's another
• 23:55
  question today what is the model nowadays?
• 23:57
  The machine learning or that's MOT would be building is a model
• 24:02
  is a prompt, but let's get to its in a moment.
• 24:06
  So. Control confidence around the
• 24:11
  building and testing, similar to what we have in software space.
• 24:16
  I'm coming from software space and.
• 24:20
  Today. I think the process is around
• 24:23
  DevOps. OK, there's constant development
• 24:27
  around it, but they're pretty stable and and and you can.
• 24:32
  Iterates develop tests. Deploy and monitor on
• 24:37
  production. Software in the pretty cool,
• 24:40
  kind of controllable, confident way and I think.
• 24:44
  This confidence and control over the process is a fundamental
• 24:48
  fundamental need. If you want also in mill space
• 24:53
  array a space if you want to truly use those models on
• 24:57
  production. So.
• 24:59
  I think the when when when you look at the large language
• 25:03
  models I see. This is even more challenging
• 25:07
  now. Because.
• 25:10
  Yes, we we see already that there are models based on those
• 25:15
  on tag PT34 on production already, right.
• 25:20
  But I would say that mostly you see it in the use cases for the
• 25:24
  use cases where it can be wrong. Or where there is a human in the
• 25:30
  loop. Sometimes human in the loop is
• 25:32
  the customer, but it is still human in the loop.
• 25:34
  Tye of fig. Because.
• 25:37
  I think that. We are to some extent with large
• 25:42
  language models in a similar times to.
• 25:45
  Two years like early deep learning times when we were
• 25:49
  initially kind of in the discovery phase we we spotted we
• 25:55
  wow we we have new capabilities and every month you have the
• 26:00
  like today this every week you have a new announcement what is
• 26:03
  that extra capabilities unlock what is.
• 26:08
  Durable Low. But.
• 26:11
  It is still I I I think that's. We'll see a huge progress here,
• 26:17
  but. The use cases where this part
• 26:21
  where such models are truly used in production today are quite
• 26:26
  limited to those who are human is in the loop to really unlock
• 26:31
  it and use it also for. Like you have in classical deep
• 26:36
  learning that you can control, you can test and see some
• 26:38
  validation metrics, you can have some level of confidence.
• 26:42
  We are using them in predictive maintenance in use cases where
• 26:45
  there is no no human in the loop, so.
• 26:49
  So I think that. We will see.
• 26:53
  Improvement like new methods, how we are going to test,
• 26:57
  validate those new models or new ways, let's call them models and
• 27:03
  I think it is. I don't know exactly what would
• 27:05
  be the solution, what would be the method here, but I see that
• 27:09
  this part is. The area where Neptune as a
• 27:14
  company because of the of the mission, we'll be involved and
• 27:18
  we'll try to like on one hand discover those ways and support
• 27:22
  those ways of testing, validating those models prompts
• 27:27
  quite likely I expect to see some level of combination hybrid
• 27:31
  with classical machine learning deep learning models.
• 27:35
  Maybe via agents. It is one visible way.
• 27:39
  So you know that today you can you can allow a model to ask,
• 27:45
  use Google search, use Google Wolfram alpha to calculate
• 27:48
  something. I also expect that an agent can
• 27:51
  be another model that would be more classical, one that would
• 27:54
  predict something or vice versa, like closer to transfer learning
• 27:59
  or embedding techniques. But, but I think that, yeah,
• 28:03
  that's when you look at the core needs.
• 28:06
  Let's have those models on production.
• 28:08
  We will need to have like ability to test those models and
• 28:12
  control them and it is we're adapting.
• 28:16
  As a company is trying to contribute so, so I think that
• 28:20
  that, yeah, that's that there would be a lot of work to be
• 28:22
  done around it for those new techniques.
• 28:26
  But. So it's one thing, second thing
• 28:28
  I still believe that the current techniques.
• 28:31
  For many use cases are are still relevant and I remember.
• 28:36
  Do you remember people are not maybe anymore are talking about
• 28:40
  them so often, but Alto mill concepts?
• 28:43
  Yeah. Oh my God, yeah.
• 28:45
  It's so funny. 20/19 was big for Auto ML.
• 28:49
  Yeah. And and there was a question,
• 28:50
  Ohh guys, maybe we won't need data scientists, right?
• 28:54
  I was asked by my submitted customers.
• 28:58
  Will they so they build ML team or it will be just out ML and
• 29:03
  yeah. And Ultra Milk today is called,
• 29:08
  I would say, HBO or HBO related techniques, right?
• 29:12
  Sure. So I think that that we will be
• 29:17
  mixing different techniques depending on the problems.
• 29:21
  If you want to call and we'll be able to cover more and more
• 29:25
  types of problems, the amount of investment in this or related
• 29:29
  fields is growing. So we will be also discovering
• 29:33
  more stuff that is doable. Speed, which is what you want to
• 29:37
  do is provide control and confidence to the people that
• 29:41
  are in charge of these models, so to speak, whether they're
• 29:44
  data scientists or whatever they might be.
• 29:46
  And you want to do that regardless of whatever paradigm
• 29:51
  we're in, whatever context we're in, it's almost Neptune is
• 29:54
  indexed to the growth of machine learning.
• 29:56
  Machine learning is going to grow.
• 29:58
  And so Neptune will grow. And you know, the product will
• 30:00
  evolve along the way, but the mission stays the same, control
• 30:03
  and confidence. And I love that.
• 30:05
  Well, there is one thing that I wanted to jump into when it
• 30:08
  comes to. Just something that you said
• 30:11
  earlier and you kind of sprinkled it into the
• 30:12
  conversation around. The experiment tracking now
• 30:16
  becoming prompt tracking and if that is going to be the
• 30:20
  evolution and I would love to hear your thoughts on that
• 30:23
  because it feels like there is a definite need for prompt
• 30:26
  tracking. But I don't know that experiment
• 30:30
  tracking is going to 100% go away.
• 30:33
  It still feels like there's a tonne of use cases for that, and
• 30:38
  so how are you looking at both of?
• 30:39
  Those. I said this sarcastically.
• 30:42
  OK. So, so again I I'm looking from
• 30:59
  jobs to be done perspective. An.
• 31:04
  So in the short from short time person let's say one year
• 31:08
  perspective and this is what I what I see and I can share with
• 31:11
  you. I'm we are seeing more.
• 31:15
  Traction and demand for experiment tracking even without
• 31:20
  large language model. I think it is side effect.
• 31:25
  Of a bit of a bigger investments into like broadly speaking I.
• 31:32
  So yes, we have a new big sum that makes everything else looks
• 31:37
  way smaller, right? But the reality is that there
• 31:42
  are a lot of synergies. So, so I think it is, it is why
• 31:47
  we see it but when it comes to experiment tracking.
• 31:52
  Let's let's discuss what it is about.
• 31:55
  And by the way, I really. Don't but we use this because.
• 32:00
  Like maybe you will need to. You will be using with
• 32:03
  envelopes. Name.
• 32:05
  We use experiment tracker because name because it is what
• 32:08
  market understands. But if you look at the jobs to
• 32:13
  be done of experiment tracker, they are.
• 32:17
  Weight behind, beyond just experimenting.
• 32:20
  Experimenting sounds pure research, no production so.
• 32:25
  I would say that's main jobs to be done.
• 32:28
  Often experimental tracker would be about.
• 32:31
  On one hand, when you're building models, you want to
• 32:34
  understand what is happening. You want to understand the
• 32:37
  building process. You want to debug it?
• 32:41
  You want to. Compare it with other
• 32:45
  experiments. So in this way you would
• 32:47
  understand like whether the model you're building is going
• 32:51
  into the right direction or not. You want to version it so you
• 32:55
  have some level of reproducibility, some way to.
• 32:59
  Maybe Sir particular model or service for a feedback and you
• 33:05
  want to make it in a way that is.
• 33:08
  That you can hand them over model over to OPS team.
• 33:12
  So from this perspective, when I think about prompt engineering,
• 33:16
  that is. Quite a different way of
• 33:19
  building models. I'm not even say.
• 33:21
  I'm not even sure that we should be calling.
• 33:24
  Prompt engineering model building process.
• 33:28
  Right. Ohh yeah.
• 33:29
  Because the model like is stateless when you think about
• 33:34
  OK, fine tuning. For instance is not available
• 33:36
  for the latest models. Is for GPD free, but GPT 3.5 and
• 33:40
  four. As far as I remember you cannot
• 33:42
  find unit. So what You're left, You're left
• 33:46
  with how you will craft prompt. Plus more, right you can you can
• 33:52
  configure agents and build that prompt in a sequential way using
• 33:57
  different models. So yes, this engineering.
• 34:02
  And. But.
• 34:05
  Here we are talking about building phase.
• 34:08
  How I would understand how does this work et cetera.
• 34:11
  So yes, I see that and it is on our road map.
• 34:15
  I see a support of prompt visualisations or chain
• 34:20
  visualisations. Integration with long chain is
• 34:23
  obvious thing, but this is just the beginning.
• 34:27
  I think that. To really support.
• 34:33
  MO, maybe not only mill. Teams that are building such
• 34:37
  models. And using the non production we
• 34:40
  will eventually would also need to think about how we are going,
• 34:44
  how we are going to support and what are the methods, how we are
• 34:49
  going to validate those prompts because today it is very much
• 34:53
  human. Pee on right.
• 34:55
  I'm looking at this problem. I'm looking at this problem.
• 34:57
  This prompt looks better. There are ways to like there
• 35:01
  are. Ideas.
• 35:03
  Let's try. Let's do this validation also
• 35:07
  with foundational models. That would be judging which
• 35:10
  which set of prompts is better. But by the end of the day, we
• 35:14
  would also need to figure out how we are testing products that
• 35:18
  would symbolise models. Be with me to We will need to
• 35:23
  think about how we are going to what, what are what, what will
• 35:28
  be the techniques we can really use.
• 35:30
  In order to update those models because those models like yes
• 35:35
  like if you are asking very. Like questions that we've
• 35:39
  generated context, maybe you don't need to update or you can
• 35:42
  just update with the new version of GPT available, but if you are
• 35:46
  building. Let's call it a solution based
• 35:50
  on Foundational Model that is doing something specific with
• 35:54
  your data. Then maybe fine tuning.
• 35:58
  Maybe it would be done by a agent that would ask your kind
• 36:01
  of smaller classical model for. Some prediction and and the
• 36:09
  prediction will be taken into consideration by the
• 36:11
  Foundational Model. So I I think, I think we will
• 36:15
  also need to be able to connect the dots what is using what,
• 36:18
  which version. So it is very much what
• 36:21
  experiment Tracker is doing today.
• 36:24
  So I think that. It shouldn't be called prompt
• 36:28
  Tracker. It will be even worse name than
• 36:32
  Experiment Tracker. I don't like Experiment tracker
• 36:34
  as well. It's not.
• 36:36
  I'm not sure somebody crafted a name evaluation.
• 36:39
  Store. Oh yeah.
• 36:42
  It sounds, to some extent close. I'm not good in figuring out
• 36:47
  names. Plus you know how it is with
• 36:49
  names. You can name something as you
• 36:52
  wish. Maybe you would be super
• 36:53
  accurate, but what is important is how people are.
• 36:56
  What's the mark like? How would the market understand
• 37:01
  That's why that's why we are we stick to experiment tracker but
• 37:05
  but I think that set of jobs to be done.
• 37:08
  At least for our understanding of experiment tracker is a
• 37:12
  little bit broader. And Prompts Prompts Engineering
• 37:17
  will be one of the techniques. Relevant.
• 37:20
  Very important. We will support it, but it is
• 37:23
  not the end. It's basically getting that
• 37:26
  whole view and knowing all of the lineage and knowing
• 37:29
  everything about it from. Exactly the.
• 37:33
  Basically I kind of look at it as the control pane and being
• 37:37
  able to sit back and say OK, I understand everything that is
• 37:40
  happening within this use case or what I'm trying to do with
• 37:44
  this. Job to be done.
• 37:47
  What? What's my end state?
• 37:48
  And getting to that end state, there's a million different
• 37:51
  things that have happened, and so I want to be able to see all
• 37:55
  of these with a quick glance if I need to.
• 37:58
  So I like. This super important, this super
• 38:01
  important if you are talking about production because you
• 38:04
  need to have confidence and kind of level of control right from
• 38:07
  technical point of view what I believe and maybe not the
• 38:11
  biggest fan of regulations, but I think some level of
• 38:15
  regulations. Will be regulations audits will
• 38:21
  be. More.
• 38:23
  Like bigger topic. In the future.
• 38:27
  Yeah. I mean, yeah, so so you know
• 38:29
  like you will need to have some level of order and control what
• 38:33
  what you've been doing and how not only from engineering
• 38:36
  perspective, but I'm motivated mostly by engineering needs like
• 38:42
  having order control process. Being able to reproduce staff,
• 38:46
  being able to compare stuff, being able to say this is
• 38:49
  better, but it is not. I feel it is better.
• 38:52
  It is better because we've performed those tests.
• 38:54
  Here are the results. It is why we are making this
• 38:56
  call. So I think this has been a great
• 39:00
  conversation. We have touched on a lot here,
• 39:03
  especially when it comes to thinking about the way that
• 39:05
  machine learning will be in the future, what kind of company
• 39:07
  Neptune is going to be to accommodate that, what the
• 39:10
  future of the experiment tracking category is.
• 39:12
  I, I kind of just, I kind of want to ask you as we wrap up
• 39:15
  this conversation, Peter, two years from now, what is your
• 39:21
  prediction or what is the future that you believe will be skating
• 39:24
  towards right? I want to get a sense of as we
• 39:27
  talked about product vision, right?
• 39:29
  Like where do you think, how do using the machine learning
• 39:32
  workflow will look like 2 years from now and what kinds of tools
• 39:35
  will people need? What's your prediction there?
• 39:39
  OK, hmm. I was not.
• 39:42
  I was not kind of prepared for this question, but that's the
• 39:46
  best way to do it. It's off the off.
• 39:48
  Yeah. I I think that we'll definitely
• 39:51
  have. Numerical methods.
• 39:55
  To test. And validates models built on
• 39:59
  that are built with or next to foundational models.
• 40:04
  So this will be kind of under control we'll we'll be, I also
• 40:10
  believe we'll be like having defined ways how we combine
• 40:14
  classical models that I I don't feel that they are they won't be
• 40:20
  relevant. I believe they will be relevant
• 40:22
  but for a specific tasks that are less generic.
• 40:28
  So you don't have a lot of public data to like OK maybe
• 40:33
  predicts predictive maintenance type of problems, recommender
• 40:37
  systems, things like that. So we'll have a ways of
• 40:42
  combining foundational models with classical models and and I
• 40:47
  think Emma, OPS, big stack or community will need to
• 40:52
  understand how to use it in a controllable, controllable way.
• 40:56
  I also believe that because of the markets, tech markets, I
• 41:00
  would say correction it is tougher for acid for leptin is
• 41:06
  we have been always building a company in a kind of marathon
• 41:10
  game in mind. So we have not spent a lot of
• 41:13
  money. So we are still very close to
• 41:15
  yeah amount of money we burn super lean, but I think that
• 41:21
  what would be good for players like Neptune that is as more
• 41:25
  point solution than end to end platform.
• 41:27
  The good thing is that we will know with whom it makes sense to
• 41:32
  integrate. So we will be able to provide
• 41:35
  more end to end solutions. Not being end to end solution
• 41:38
  because for us it was always a problem.
• 41:41
  If you are selling to enterprise like yeah we are components
• 41:44
  doing experiment tracker, yes great.
• 41:46
  But man I have a problem like I need to provide text to my to my
• 41:52
  e-mail organisation and so I wanted to have.
• 41:57
  OK, sure, here's our end to end solution.
• 42:00
  Like, here's the whole solution. Neptune is here.
• 42:03
  Here you can pick up one of the partners, but it was problematic
• 42:07
  because this market was so crowded.
• 42:11
  We didn't have money to to integrate with everybody, right?
• 42:14
  So. So I think we'll have some level
• 42:17
  of natural selection. There will be less players, less
• 42:21
  leaders, but more clear categories that will be well
• 42:24
  integrated. So on one hand I see a lot of
• 42:27
  development in end to end platforms, but I am not a big
• 42:31
  fan. I don't see a lot of end to end
• 42:34
  platforms in software space. I think we'll have more clear
• 42:38
  categories of point solutions that are well integrated OI
• 42:42
  think this market is I would call it's way more mature
• 42:46
  market. On the other hands, I'm super
• 42:49
  curious to see what would be like.
• 42:52
  What is the cup or where is the selling for capability of the
• 42:59
  current methods and what are the new methods that are coming?
• 43:03
  So what is the what is the true potential here?
• 43:06
  But this parts I cannot predict. Two years ago, I wouldn't be
• 43:10
  able to predict. I didn't know, even though I was
• 43:13
  following this space very closely, that we will be able to
• 43:15
  do what we are able to do now with foundational models.
• 43:19
  Yeah, I. I love the way that you put that
• 43:22
  and it's so funny. I just want to say at the same
• 43:25
  time that we're talking about machine learning.
• 43:27
  We still have problems with Wi-Fi.
• 43:29
  So I don't know if the whole world is going to be solved, but
• 43:33
  Demetrius and I during the course of this conversation had
• 43:35
  some Wi-Fi problems. So maybe hey, some real world
• 43:37
  things to solve. But I think that the vision that
• 43:40
  you lay out there in terms of knowing exactly where you'll be
• 43:43
  able to embed and and and connect in with other sort of
• 43:48
  infrastructure providers to create the sort of end to end
• 43:50
  ecosystem. And the fact that we'll have
• 43:52
  those numerical methods as you said that that give us the
• 43:55
  fine-tuned control and the precision that we need in order
• 43:58
  to scale these systems, I just love that.
• 44:00
  And I've got to say it's very good for not having thought
• 44:03
  about this before and honestly it's probably you probably have.
• 44:06
  So I'm going to call a little bit of cheating.
• 44:08
  But Peter, thank you so much for joining us for making it a great
• 44:11
  conversation, for sharing us your product vision, for giving
• 44:14
  us a preview into the way in Neptune's thinking about the
• 44:16
  future. I'm long Neptune and this is it
• 44:19
  was a great conversation. Thank you for having me.
• 44:20
  Thank you guys. Really happy to be part of that.
• 44:24
  Biggest Emma loves podcasts. I think even bigger than ML OPS
• 44:29
  depending on the name. Podcasts out there.
• 44:32
  Thank you guys. There we go.
• 44:33
  Awesome. There's any last words here?
• 44:35
  Yeah, this has been great. Perfect.
• 44:38
  Thank