Latest / Agentic Conversations (formally mlops.community) / Evolving Workflow Orchestration // Alex Milowski // #291

Transcript

Evolving Workflow Orchestration // Alex Milowski // #291 · Agentic Conversations (formally mlops.community)

• 0:00
  I'm Alex Bilowski, there's lots of ways to pronounce my name
• 0:02
  which is a fun fact, but could be Bilowski to Miyowski too if
• 0:07
  you want to be the Polish version.
• 0:09
  And I'd usually take my coffee any way I can get it.
• 0:12
  I love a flat light but it's very hard to find in the US,
• 0:15
  like proper flat light so I got it's cooked on them in
• 0:19
  Edinburgh. Welcome back to the ML OPS
• 0:22
  Community Podcast. This is the definite guide on
• 0:26
  workflows, AKA Dags, AKA pipelines.
• 0:30
  We get into the nitty gritty of why they're valuable, how you
• 0:35
  can use them. We're talking workflows within
• 0:38
  workflows, workflows that are dependent on other workflows.
• 0:42
  It's just a constant deep dive on what they are, why they're
• 0:48
  useful, and how you can best take advantage of them.
• 0:53
  My man Alex did a whole survey of different tools that are out
• 0:59
  there. He covered 79 of them, and we'll
• 1:04
  leave his blog post to all of the insights that he gained from
• 1:10
  this survey of all the open source workflow tools on GitHub
• 1:16
  in the description so you can check it out.
• 1:19
  Let's get into the conversation with Alex Workflow Systems and
• 1:31
  what those mean, what it is, what got you interested in where
• 1:35
  are we coming from with that? Because you've got lots of
• 1:39
  thoughts on it. And there's this really cool
• 1:42
  like summary or survey that you showed me.
• 1:46
  So I want to dive into that. But let's just set the scene
• 1:49
  with Workflow Systems. Sure.
• 1:51
  The idea of a workflow, I, I had to look this up and it goes back
• 1:55
  to really the 1920s, which is called Pilot.
• 1:58
  The, the term itself comes out of people looking at process
• 2:02
  engineering for manufacturing and other kind of business
• 2:06
  contacts. So that makes sense, right?
• 2:08
  And it is how we use it today. But the real sort of genesis of
• 2:12
  the current generation, I think comes out of more like rule
• 2:16
  based expert systems. The generation maybe some of the
• 2:20
  90s where people were trying to build instead of build writing
• 2:25
  code that implement business processes.
• 2:28
  They have rules and the rules would it would use a rules
• 2:31
  engine and the rules engine would tell the application or
• 2:33
  whatever, what's the next thing to do?
• 2:36
  And so there's this sort of implicit idea of a chain of
• 2:39
  things that somebody has specified him.
• 2:42
  What's the first step in there? If we that successful and maybe
• 2:45
  some criteria and then what's the next thing the average user
• 2:49
  might think of those as our sort of person might think of those
• 2:51
  as like a filter, But it's more computer science he taught.
• 2:55
  It's a it's a big graph. It's A and inside the graph is a
• 2:58
  bunch of steps and the steps are basically tasks that do things
• 3:01
  and manipulate data and produce results or have side effects.
• 3:05
  And then when 1 successfully completes some downstream task,
• 3:08
  it's involved. And that is the workflow by
• 3:11
  itself. And there have been processing
• 3:13
  with systems developed post rules engines world that think
• 3:17
  of that more like a graph and they think it's more human
• 3:19
  intuitive to draw out that flow chart of what they want to do.
• 3:24
  And and there's a whole sort of other side of this world that's
• 3:27
  not used in machine learning as much, but this whole business
• 3:30
  process modeling. And there's actually standards
• 3:33
  around that. There's a thing called BPM and
• 3:36
  there's a business process modeling notation.
• 3:38
  And those are very cool things themselves that the people use
• 3:41
  to describe processes that they use inside their businesses for
• 3:44
  all kinds of purposes. And the having an exchange
• 3:47
  format notation for that is great.
• 3:49
  But that's sort of, you know, the, I think bifurcation here of
• 3:54
  those kinds of systems that were built for sort of business
• 3:57
  processes in general versus how we have used them in data DevOps
• 4:02
  and data engineering, MLA operations and so forth.
• 4:06
  And then there's a kind of a small category of things that
• 4:09
  are workflow engines that are embedded in applications
• 4:11
  themselves, which kind of are all over the place.
• 4:17
  And so my interest was to look at the sort of more broader
• 4:21
  context because revisiting things I've done in the past and
• 4:24
  all this new stuff and what is out there.
• 4:26
  And I spent some time on GitHub looking at, I found some people
• 4:31
  had some great lists and I found some best ways to search for
• 4:34
  these things. And I basically built a big
• 4:36
  spreadsheet of workflow systems and features that they have and
• 4:41
  are trying to sort out these different systems and what where
• 4:44
  they when did they start or are they active and features do they
• 4:47
  have versus others. So.
• 4:50
  Presumably the majority of them are geared towards technical
• 4:54
  users. Yes, that's just said there's a
• 4:56
  bifurcation that the the business side of it that do this
• 4:59
  more of the business process modeling.
• 5:02
  They have nice interfaces. They're meant for you're a sort
• 5:05
  of average user inside a corporation or someplace to to
• 5:09
  be able to describe a process. Some of them have a sort of a no
• 5:12
  code aspect of them. In the end of the day, it's
• 5:15
  still a very technical thing to to draw one of these big
• 5:19
  diagrams, even if you have a beautiful tool to do it with.
• 5:22
  You have to know a lot and you have to think a lot about like,
• 5:25
  how does we actually do this? And sometimes they're replacing
• 5:30
  or modeling things that are done by humans.
• 5:32
  And I have a whole story about that from last company.
• 5:34
  I was working it with scientists, Biden, how I was
• 5:38
  trying to re react to use the this, this BPN notation just to
• 5:42
  help them like draw us a picture.
• 5:44
  And I needed a standard way to draw a picture of what they were
• 5:48
  doing. And so that business process
• 5:50
  modeling notation in the scientific context was actually
• 5:52
  a useful tool. Right.
• 5:55
  It is funny that you talk about how we can represent it as
• 5:59
  graphs because I've heard this before in this last year when we
• 6:03
  had the AI Quality conference, one of the speakers who actually
• 6:08
  is one of the creators of Docker, Solomon, he talked about
• 6:12
  how everything is a graph. And then my friend that was
• 6:15
  sitting next to me, David, he said, wow, I never thought about
• 6:18
  that. It's yeah, everything's a DAG.
• 6:20
  You can really represent anything as a DAG kind of.
• 6:24
  And so it's almost like thinking about it like that.
• 6:27
  Yeah, graphs were very, super useful notation.
• 6:30
  They're also can get messy really quick my and this is
• 6:33
  where you need tooling and so the notations and so the the
• 6:37
  depends on what you're doing with it.
• 6:39
  I think the what I found interesting and we stumbled on
• 6:43
  this BPN and I had not been following that area for a while,
• 6:47
  but I was, we were struggling with how do we draw a picture of
• 6:51
  these process engineering flows because we want to automate
• 6:55
  this. We have machine learning, we
• 6:56
  have robots, we have humans, right?
• 6:58
  And they're all part of this process.
• 7:01
  And we have, we have different interactions there.
• 7:03
  And when a person does something in the lab and they take a, they
• 7:06
  literally have a plate with wells in it and, and oculents
• 7:10
  and stuff in it and they're putting it inside a lipid handle
• 7:13
  a robot or they're putting inside an incubator robot.
• 7:16
  We need to know what that task that the human did before the
• 7:21
  robot is told to do their task before the machine learning is
• 7:24
  told that, you know, with the post data to do its today.
• 7:27
  And so that modeling of that whole process let us build tools
• 7:32
  and user interfaces to make the lab more efficient.
• 7:36
  And that's one of my sort of takeaways is from that
• 7:39
  experience was that there's what we do in ML OPS for like just
• 7:44
  the part, the technical part that we're concerned about.
• 7:47
  And then there's how it's used in the bigger organization.
• 7:49
  And that's also a workflow. And so it's like workflows
• 7:52
  inside workflows. And you know, how, where you
• 7:56
  slice it, it, it's important to how the sort of result that
• 8:01
  you're trying to trying to get. So in this case, we're trying to
• 8:03
  make the, the processes that the lab used more efficient to make,
• 8:09
  to get more throughput to the lab as well as to get all of the
• 8:12
  interesting results from the computational side of it.
• 8:16
  And so writing that workflow down as a whole thing with all
• 8:20
  the technical bits in there too. So everybody understood all the
• 8:24
  parts was a was a challenge, right?
• 8:27
  So having a notation was first like, how do we communicate
• 8:31
  this? How do I get a literally a
• 8:34
  drawing doesn't have to turn into code, but just a drawing
• 8:36
  that everybody says, yes, that's what we do, right?
• 8:39
  Yeah. And from all sides, right.
• 8:40
  And even if their eyes glaze over for a part of it, that's
• 8:43
  not their thing. We all have have consensus and
• 8:45
  we're building all the same thing.
• 8:46
  So that we stumbled across this mostly because there was a
• 8:49
  really cool web-based tool out there from a company called
• 8:52
  Commandas. But they have a open source
• 8:55
  thing and runs in the browser and you can drag and drop things
• 8:58
  and build the whole workflow and then it spits out a artifact.
• 9:02
  Nice BBNN notation file format. So theoretically down the road
• 9:09
  you could do something with that.
• 9:10
  Our in our case it was simply just a diagram that we could use
• 9:13
  as a communication piece as part of our sort of internal
• 9:17
  technical adaptation. But it was a good starting point
• 9:20
  for that, just drawing a picture of what are we implementing with
• 9:23
  our workflows to steps. You talk about the way that you
• 9:27
  slice it and being able to look at the workflows almost like
• 9:34
  from infinitely zoomed out to infinitely zoomed in and all of
• 9:39
  the different layers of workflows involved in each time
• 9:43
  that you're zooming in. And then where one workflow
• 9:46
  starts and one stops and who owns which workflow.
• 9:51
  And then you're getting into technical workflows and versus
• 9:54
  non-technical workflows. And do you have any experiences
• 9:58
  on what is useful in that regard?
• 10:03
  Like how to slice and dice these?
• 10:06
  Yeah. I think the one of the
• 10:08
  challenge, one of the things is to maybe not try to think of it
• 10:11
  as there is the one workflow. So I think the on the outside
• 10:16
  there's the process of your business, whatever it might be.
• 10:19
  So if you're a very in silico technical kind of organization,
• 10:23
  like you have a digital product, right, and you're using machine
• 10:25
  learning, there's the aspects of the tech internal piece of how
• 10:31
  that model desk inference or is trained or is evaluated.
• 10:35
  And that's a very technical workflow.
• 10:36
  That's something that a small team, sorry, somewhere in your
• 10:39
  organization really understands well.
• 10:41
  And then it's a black box and it's a black box in a larger
• 10:44
  workflow. That is how your organization
• 10:48
  uses that, how they make decisions around it.
• 10:50
  So train a new model. The output of that, the whole
• 10:54
  process might be not just the model, but how good is it at a
• 10:58
  particular task. Somebody has to make a decision
• 11:01
  about about what they do with that decision could be
• 11:03
  automated. If it passes certain criteria.
• 11:05
  We we put it into some kind of production track.
• 11:09
  It might not be, it might be there might be a human who has
• 11:11
  to go in there and make a decision and start the ball
• 11:14
  rolling. That's part of a bigger process
• 11:17
  and that's over a business process workflow.
• 11:19
  And so I think there's these, there's opportunities here to
• 11:22
  have a layer model where you can be using and maybe using
• 11:26
  different technologies, different workflow systems,
• 11:28
  right? But still they're interacting
• 11:30
  because one is 1 is using the other.
• 11:32
  And in the case of my last couple, the auto workflow of
• 11:37
  what is the scientist labs doing versus the technical bits of
• 11:43
  each tool that they're using. That was just a paper
• 11:46
  discussion, right? It's a diagram, it's a part of a
• 11:48
  documentation. It's so that we all understand
• 11:51
  what we're doing. But maybe the ultimate goal is
• 11:54
  that it's executed by some system, but there's a long term
• 11:58
  goal versus the short term goal, which is so now if you dig into
• 12:01
  the boxes in this thing, how do we accomplish that task?
• 12:04
  Maybe there's a big procedure, it's a wet lab thing, maybe it's
• 12:07
  a robot, maybe it's a whole machine learning workflow that
• 12:10
  runs a bunch of code and manipulates a bunch of data.
• 12:13
  But so there's you going to have, you know, you can draw the
• 12:16
  workflow out and you can execute it and you can have a system
• 12:20
  that actually implements it. And I think those are useful
• 12:23
  architectural sort of ways to decompose the problem.
• 12:27
  And then you can choose where you implement your and where you
• 12:29
  spend your time and monies implementing a system, right?
• 12:32
  And there are choices for each of those things.
• 12:34
  That's what's cool about this is you can go full on like I got
• 12:37
  work for everything and I've got a system for everything and
• 12:39
  there are tools for those things.
• 12:40
  That's the cool place where we are at this point in time, which
• 12:43
  was not true even like a decade ago.
• 12:47
  Yeah, it does feel like if you are able to understand the
• 12:52
  different workflows and what happens after your workflow
• 12:56
  ends, in a way you are setting yourself up for greater success.
• 13:02
  Because if we take that example of, OK, there's your little
• 13:06
  piece, which is the model, and then exposing the model to the
• 13:10
  greater organization. That's great.
• 13:14
  And you can be optimizing the model for things that you think
• 13:17
  are cool. But if you really know what
• 13:20
  happens after you expose it to the organization and how other
• 13:24
  teams are using it or what they're looking for when it
• 13:27
  comes to that model, then you're optimizing for something that's
• 13:33
  greater than just what you think is useful and your understanding
• 13:37
  how it's being used in the greater context.
• 13:40
  Have you seen that being a a case that when you notice all
• 13:46
  the different ways and dependencies that are being
• 13:48
  built from your specific workflow, you're setting
• 13:54
  yourself up for more success? I think that it depends on the
• 13:58
  scope there. So I think that workflow
• 14:01
  systems, when they are properly handled, however, whatever kind
• 14:06
  of technology you choose, so you have to support them, they have
• 14:08
  to be vibrant and they have to meet the user's needs so they
• 14:12
  can Stitch Fix. You know, when I was there,
• 14:14
  there was a higher than 40 data scientists running stuff for the
• 14:19
  organization. Some of these things were the
• 14:21
  sort of back house types of things.
• 14:23
  Some of these things are more daily things that happened that
• 14:25
  populated their systems internally and they were really
• 14:28
  critical and some of them were research work and so forth.
• 14:31
  And so all over the place. The ability to describe the
• 14:37
  steps of things and interact with it was airflow underneath
• 14:41
  or wasn't, but they didn't actually interact with the
• 14:42
  outflow. They had ADSL that let them the
• 14:45
  term we should define the domain specific language, but they have
• 14:48
  a way to describe the workflow in the DSL and then they could
• 14:51
  give it over to the system and they would run it through by
• 14:54
  underneath with Airflow. And then they had a way to
• 14:56
  execute the tasks on their big serve batch system.
• 15:00
  All that complexity was hidden from those 140 users, so they
• 15:04
  didn't have to become experts in that technology.
• 15:06
  And it did lots of good things for them like the task executor
• 15:09
  auto learned stuff like, oh, your task needs more memory, so
• 15:12
  we're going to retry it with more memory because it ran out
• 15:14
  of memory and it learned the right parameters for the user.
• 15:18
  So again, they didn't have to be extra some deployment.
• 15:20
  You can do lots of cool things with workflow systems that and
• 15:23
  you can. So that gives you that they're
• 15:26
  more productive, right? They are less frustrated until
• 15:28
  the system breaks and then they're frustrated with it.
• 15:30
  But happy users are assigned like this person sounds.
• 15:32
  Yeah, and you so I, I think that I've seen that kind of success
• 15:37
  what I I haven't seen yet and maybe tell me, but tell me I'm
• 15:42
  wrong. I haven't seen a jump silos,
• 15:44
  right. So that's a data science
• 15:46
  organization inside a big company that your tools used
• 15:49
  internal to that part of the organization.
• 15:52
  But you don't see nesting of workflows as much where you see,
• 15:55
  and this is now a part of a bigger workflow.
• 15:58
  I think those are more, those processes are more bespoke in
• 16:01
  terms of if the data that's used by the output of the machine or
• 16:05
  workflow, the the model that was updated is used by some of our
• 16:09
  system by some other connection that is specific to how we
• 16:13
  deploy that thing. And the idea of a workflow using
• 16:17
  a workflow and, but they're different parts of the
• 16:19
  organization using workflow systems.
• 16:22
  I haven't seen that sort of in, in real life.
• 16:24
  I'd like to see it. I think that's a vision.
• 16:27
  Yeah. We describe our organizations as
• 16:29
  process. Like I like to think of as
• 16:32
  process engineering, right? And we can take that down.
• 16:34
  And but there's a term I'm borrowing from, like
• 16:37
  manufacturing, where, you know, everything has to follow a
• 16:42
  procedure. And we describe what this
• 16:43
  procedures are and they fit together like puzzle pieces.
• 16:46
  And I think business and digital context can work the same way,
• 16:50
  but you have to do that process engineering and it's hard.
• 16:52
  And so that people kind of skip that hard step and but there's a
• 16:56
  payoff. People will use these systems.
• 16:59
  And, and so I think we see that with data engineering and ML OPS
• 17:03
  and, and operations where there is a payoff, right?
• 17:08
  Because there's all this complexity about how we do our
• 17:10
  task. And we could hide that in the
• 17:12
  system and make the end user who is a data scientist or machinery
• 17:16
  engineer or some other kind of data person data engineer.
• 17:20
  They're they're just more productive because most those
• 17:23
  details are left to a very small handful people in a corner
• 17:27
  making that workflow system work for them.
• 17:33
  Why do you think it is hard, you mentioned to do this process
• 17:38
  engineering? Is it just because it's time
• 17:40
  consuming and cumbersome and there's a lot of friction there
• 17:43
  to describe the systems and explain the processes and and
• 17:48
  it's not really like you get, you can't just do it and it gets
• 17:53
  auto documented for you really, right?
• 17:56
  I, I think the, my last company experience was, is interesting
• 18:00
  because my main users were internal, they were scientists
• 18:04
  and they are, this is not a world view that they're used to
• 18:09
  thinking about in terms of like, we're going to describe our,
• 18:12
  what we're doing in our lab as a graph of things of tasks and the
• 18:17
  rules to writing like procedures to operating procedures.
• 18:20
  And it's a big document with hierarchies of things.
• 18:23
  And next you do whatever. And next you talk to this
• 18:25
  machine and set up these protocols.
• 18:27
  And so the idea of looking at it as a graph of tasks and how they
• 18:33
  interact and how you interact with that and annotating it with
• 18:36
  here's the data that I need. Here's the pre steps that have
• 18:39
  to happen before I can start this task.
• 18:41
  Here's the controls I have. Here's the failures that could
• 18:44
  happen in this situation, that level of they're just not used
• 18:48
  to writing out. And you can understand that
• 18:51
  because that's not usually their, it's not useful to them
• 18:55
  in their data because they know how to handle those situations.
• 18:58
  And then they're familiar with equipment, they're familiar with
• 19:00
  the work that they're doing. So what they've been trained to
• 19:02
  do, that's what they've been doing in their career.
• 19:05
  But it doesn't when you try to scale, and this is where the
• 19:08
  problem comes up. When you try to scale it, you
• 19:10
  try to understand more about the metrics of it, the data
• 19:14
  artifacts, the other things you might be able to get on the
• 19:16
  system. Then you need that process
• 19:19
  engineering to understand that I'm going to automate something.
• 19:22
  You have to be able to draw me a picture of what you do and tell
• 19:25
  me all the facets of it. And so that's just a hard
• 19:28
  conversation to have. And so I had varied success when
• 19:32
  I did that with these. Some people really pushed back
• 19:34
  hard. They're like, I don't know why
• 19:35
  would I have to do this? Here's the thing I'm doing,
• 19:38
  here's the procedure I have that works fine for us.
• 19:41
  And others were like, sounds interesting.
• 19:43
  The diagram looks interesting, but I don't quite understand it.
• 19:46
  And then when they engage on it at different points of time, so
• 19:49
  they all of a sudden they're like, oh, OK, I see.
• 19:51
  I can see some value in this as a maybe as a way to document
• 19:56
  what we're doing in a different way.
• 19:59
  And that's a great opener, right?
• 20:00
  If you see value in drawing the picture and I can take the
• 20:03
  picture or one of these tools and do something with it,
• 20:06
  actually build a system, that's great.
• 20:09
  And so I still had that sort of cool like.
• 20:12
  Range of responses for people. So I think that is the challenge
• 20:15
  that organization has to be able to attach some kind of a value
• 20:19
  to it. What are we getting out?
• 20:20
  What's the ROI of doing all of this process engineering?
• 20:24
  And so that's why political are like manufacturing things makes
• 20:28
  total sense that they do this kind of engineering because I
• 20:31
  have the efficiency, quality measuring stuff that's all about
• 20:34
  manufacturing. But when you're when things are
• 20:37
  much more I wouldn't call them boutique, but they're that's
• 20:40
  probably not the right term, but basically when they they're
• 20:43
  highly trained people and they're they're building these
• 20:46
  things are running a lab, they're running the business
• 20:49
  side of your organization. They that sort of technical
• 20:53
  benefit and you're asking them to speak this weird language.
• 20:57
  Where's the benefit of that? And so you have to lead them
• 20:59
  with like where's. And so I, I always like to build
• 21:01
  demos, right, or prototypes or find some exemplar that's going
• 21:05
  to give them like the why of by what I do this, which is where
• 21:09
  we started with my last company and others were like, we've got
• 21:12
  to build something that's got a captivated value somebody can
• 21:15
  see. And then there's an
• 21:17
  institutional investment in in going beyond that.
• 21:20
  Yeah, I noticed it with just this podcast.
• 21:23
  For example, I had a friend tell me really early on, you're
• 21:29
  spending too much time on that podcast.
• 21:32
  Why? What is your favorite part about
• 21:34
  it? What do you like doing?
• 21:36
  And then let's figure out how we can automate the parts or get
• 21:41
  someone else to do the parts that you don't like doing
• 21:44
  because there is a lot of intense things that you get from
• 21:49
  creating podcasts. And like you said, like I was
• 21:52
  doing one a week and feeling overwhelmed because there's so
• 21:56
  much that goes into it. I wasn't, I was dropping the
• 21:58
  ball on a lot of stuff. So I recognize that.
• 22:02
  Early in the work system. Well, I did.
• 22:05
  So I was lucky enough because my friend told me, hey, sit down
• 22:09
  with my buddy. He's really good at this type of
• 22:13
  thing and recognizing where you plug in and where you don't need
• 22:18
  to plug in. And so I sat down with a guy and
• 22:21
  he just said, So what do you do First I say if I find the guest.
• 22:25
  And then what do you do once you find the guest?
• 22:26
  I ask if they want to come on. OK.
• 22:28
  And then what do you do? And we just went through that.
• 22:29
  And what if the guest says no? Or what if the guest says this?
• 22:32
  And, and three weeks later, I had a very in depth flow chart
• 22:37
  and it helped me so much because it was the blueprint.
• 22:41
  And so I'm wondering, have you seen any trends in the ML world
• 22:46
  taking a bit of a left turn of or what are the most interesting
• 22:50
  trends that you've seen in workflows for machine learning?
• 22:55
  I think that one of the interesting trends here is that
• 22:58
  the the workflow systems in the last, I would say decade have
• 23:01
  really grown up and it is not a 10 years ago.
• 23:06
  It was more of a niche thing like my will be do this thing.
• 23:08
  It's weird. I can just write all the code or
• 23:10
  I'll write a bash script. Now it's much more common
• 23:14
  practice that, oh, you've gotten to this point from building
• 23:18
  whatever your prototype is to you need to have a workflow
• 23:22
  system. Here's your choices of systems
• 23:24
  out there that people are using. And there are older ones like
• 23:27
  Airflow and there are newer ones like Metaflow, everything in
• 23:31
  between. And so you, and then you pick
• 23:33
  your technologies and things. And so I think that is that sort
• 23:36
  of change to this is not a hard decision like you have to
• 23:40
  convince people. And no, it's like a yes, people
• 23:43
  use word plus systems to train to, to do inference, to do all
• 23:48
  kinds of tasks for them. And you should have one, right?
• 23:51
  And what's your deployment of social if using Kubernetes,
• 23:53
  there's all kinds of choices, other things.
• 23:55
  There's ones that do it more like a service orchestration and
• 23:59
  you pick, pick your sort of amongst the menu of things and
• 24:02
  there are SAS services and there are things that you deploy.
• 24:04
  And so I don't think that's the great thing is that we've moved
• 24:07
  from, it's maybe a hard sell in a corner to this is just
• 24:12
  standard practice. And, and I think the, there are
• 24:15
  a handful of companies that are doing this as a business, which
• 24:18
  is great. There's a lot of open source
• 24:20
  here, a lot of even the ones that have a SAS system, but the
• 24:23
  core technology is open source, which is what I did.
• 24:26
  That's why I went to get help. It was easy to find a long
• 24:30
  history of these things over the last actually couple decades of
• 24:34
  people building these systems, some of them done active and
• 24:37
  some of them have gone stale and are no longer active projects.
• 24:41
  And, and, but they do like they, they're in different categories.
• 24:44
  So there's the whole business process stuff, which we've been
• 24:46
  talking about a lot about the high level and and then there's
• 24:50
  just other categories. And what I did by Survey Act
• 24:52
  made the sense out. There's things for the business
• 24:54
  processing things with a generic aspect of it.
• 24:57
  And then there's things that were specifically for science.
• 24:59
  They have their own challenges there of PC computing.
• 25:02
  And then there's the sort of challenge that starts with data
• 25:05
  engineering. And then there's the data
• 25:07
  science in ML on the side of it. And then there's a little bit
• 25:10
  around operations. And so you can see the sort of
• 25:13
  newer generation of tools came out of that sort of data
• 25:17
  engineering side and then grew into data science in ML.
• 25:20
  And then a little bit of offshoot for operations, which
• 25:22
  is more like it's a management, how do I add a new node to my
• 25:26
  Kubernetes cluster? How do I install software across
• 25:29
  a bunch of machines, etcetera, etcetera.
• 25:32
  But the same kind of problem workflows, but applied in the
• 25:34
  operations context. And of the the smallest I79
• 25:39
  systems I looked at out there, little like 46 percent is
• 25:43
  business process automation and the rest of it is there.
• 25:47
  And the other, but the other big chunk is the sort of data
• 25:49
  science ML side of things, which is about 22%.
• 25:54
  And and so that's that's a trend, right?
• 25:56
  Of like these systems are growing and they're active and
• 25:59
  there's not like we're not using the business process stuff.
• 26:02
  That is a whole healthy world, right?
• 26:04
  And there's they are growing as well.
• 26:07
  And but the focus on ML OPS, which is obvious in lots of
• 26:13
  contexts, right? But these are multi step
• 26:15
  processes and the business where workflow system of a certain
• 26:19
  sork and shine. And so there's active and I, I
• 26:21
  spent some time looking at, I actually went in and said, when
• 26:26
  was this project created and, and making sure is it still
• 26:29
  active? And those are two dimensions.
• 26:31
  So because some things are personal hobbies, some things
• 26:33
  are products that kind of came in and maybe somebody abandoned
• 26:36
  it, the company went on a business or something.
• 26:37
  There's lots of reasons why things stopped getting
• 26:39
  developed. I guess we're usurped by a new
• 26:41
  thing. And if you look at that, the
• 26:43
  business process stuff is like the oldest repos out there you
• 26:46
  can find of various products that goes back to almost like
• 26:49
  2000 and five, 2006 somewhere in there when they're still
• 26:52
  actively new projects being created new things up until we
• 26:56
  sometime in the last year. So people are still innovating
• 26:59
  and building new things in the business side of things.
• 27:02
  Same thing is, but you go look at data science ML, it's more
• 27:05
  like 2015 and then from there and so actively till through
• 27:11
  last year and it's shifts back a little bit for data engineering,
• 27:15
  science has had its heyday from like 2000s to the 2015.
• 27:20
  There's lots of reasons for that.
• 27:21
  They're still actively being used, but they saw the ones that
• 27:24
  saw a very specific, this is my take, like very specific HPC
• 27:29
  problem like I am doing some massive model.
• 27:32
  They're still in use, right? And they're, whereas the things
• 27:36
  that are more like machine learning, data science, data
• 27:39
  engineering, they have new choices, right?
• 27:42
  They don't have to use these other systems.
• 27:44
  And I think there's a, there's a bifurcation there of use cases.
• 27:49
  So it's, so it's, it's interesting to see that there
• 27:52
  are different trends here, but they're also in these different
• 27:54
  sort of columns of use, right? It's fascinating that you break
• 27:59
  down like the data engineering, the SO besides the business side
• 28:03
  of the house, but if we're looking at the technical stuff,
• 28:06
  I can probably rattle off 3 or 4 data engineering E ones.
• 28:12
  When it comes to the most popular, you've got airflow
• 28:16
  that's proliferated everything and most people are using that.
• 28:20
  And whether or not they like using it is another story
• 28:24
  because it's been around the longest and it's had the most
• 28:28
  adoption. And then you have the like the
• 28:31
  mages or the Dags. I think it's Dagster and the
• 28:36
  prefix out there that are attacking it in different ways
• 28:41
  and almost like the workflow, data engineering workflow, 2
• 28:46
  point O type of thing. I would say because they're a
• 28:50
  bit more new and they're taking a different approach to things.
• 28:54
  And then in the ML world, you have the Zen MLS and the meta
• 28:58
  flows like you said. And even flight, I think is
• 29:01
  another one that is, is in there.
• 29:04
  And those are all fascinating because they're going after the
• 29:10
  ML specific type of use cases. And then the DevOps world,
• 29:14
  you've got like your Argos and maybe you could consider cube
• 29:18
  flow in that world that kind of plays in the ML world, DevOps
• 29:23
  world. So what when I was surveying
• 29:26
  this, I had a whole column and this is a spreadsheet.
• 29:30
  I was like, yes, knowing and trying to put categories and
• 29:32
  things, but I had a whole column about machine learning AI.
• 29:35
  And one of the most challenging, because I'm looking at the
• 29:38
  documentation and looking at the code and the repo is everybody
• 29:43
  is, was going to add to project is adding MLAI to their
• 29:49
  documentation as we do this, right?
• 29:51
  And and so I marked those people as neutral.
• 29:56
  And even if I found actual evidence of here is we have
• 30:00
  tasks for it, we have examples. Here's a workflow that does
• 30:03
  whatever and that that shows that you can actually do it.
• 30:06
  Then I marked it as a yes. And I think that's, there's a
• 30:09
  nuance here, which is that any of these machine, sorry, any of
• 30:14
  these workflow systems that have some kind of a model of a task
• 30:18
  executor, a lot of the Airflow included in this, they can do
• 30:24
  advanced machine learning workflows just fine.
• 30:26
  And because that task executor could be some complicated thing
• 30:31
  running on Kubernetes, it could be some other system that you're
• 30:33
  interacting with. It could be an inference
• 30:35
  endpoint that you're using. It's some inference provider
• 30:38
  like base 10. Those are all, they all have
• 30:42
  that capability. It's but I think the, the
• 30:45
  challenge here is, and that's true for the people who have
• 30:49
  business or historically also say on an older product, but
• 30:52
  it's more mature and now they're saying, Hey, we can do machine
• 30:55
  learning and AI. It's probably true there as
• 30:58
  well. It's this question of how easy
• 31:00
  is it for the practitioner to use right to actually do that?
• 31:04
  Do they, even though they said it, am I, do I have to do all
• 31:07
  the heavy lifting to make it happen?
• 31:08
  Or do you have infrastructure for me?
• 31:11
  Do you have examples? Do you have documentation?
• 31:13
  Have you, have you thought through the nuances of what I
• 31:17
  need, I'd say, say a model training pipeline and how do I
• 31:22
  get all access to assets and GPU's and things that I need?
• 31:26
  Or is that stuff I had to figure out even though I'm using your
• 31:29
  workflow engine? And I think that's the
• 31:31
  differentiator. And some of the newer systems
• 31:35
  also are kind of code or new, they're on that infrastructure
• 31:38
  as code track. And so if you're going to
• 31:40
  describe your workflow, use Python annotations and you don't
• 31:44
  have to deal with the DSL and all these other things.
• 31:47
  You just write code. And, and that's a trend right
• 31:50
  now. Everything is code and we just
• 31:51
  write it in Python. We use annotations and that's
• 31:54
  works for some people quite well.
• 31:55
  And there's nothing wrong with that, but it's not the only way.
• 31:58
  And so all the systems like Airflow, you, you know, there's
• 32:03
  also a differentiator there that the, the DAG is stored in a
• 32:07
  database, right? And so there's not a necessarily
• 32:09
  a representation of it other than to talk to the system.
• 32:14
  And there's a bunch of systems that work like this.
• 32:16
  And, and then some things have DSLS and that's it's a YAML
• 32:19
  file, it's a Jason file. It's a something else custom
• 32:23
  language. And then you write in that DSL
• 32:25
  and some things have just code, right?
• 32:27
  And that's why I tried to make a differentiation in the analysis
• 32:31
  of these different systems and looking at what, what, how do
• 32:34
  you interact with these things? And the trends for data science
• 32:37
  in ML is more cooked, right? Less annotation, your parents,
• 32:41
  less DSLS, but maybe we should talk about yourselves at some
• 32:44
  point. But but I think the and the
• 32:47
  older trend it has been that there's a serialization format,
• 32:51
  there's a thing you could author, there's an artifact that
• 32:53
  is the workflow itself, right? And it's, it's coded in Jason
• 32:56
  XML, YAML, some custom language. And what's a piece of code
• 33:01
  itself that you can check in somewhere, but it's not Python,
• 33:05
  it's something else describes all the metadata around it and
• 33:09
  treated as such. And that has it's use cases,
• 33:12
  right? Yeah.
• 33:13
  But the trend is from ML is away from that I think right now.
• 33:17
  I don't let that stick. You mentioned to me before we
• 33:20
  hit record that it feels like everything is moving towards
• 33:24
  code in this infrastructure as code world and the big question
• 33:29
  there was, is that good? Yeah, I have mixed feelings
• 33:35
  about that. I understand how it is when
• 33:38
  you're writing something that's very compact and and it does a
• 33:41
  very technical thing and it's all sitting there in Python And
• 33:44
  it's a bunch of Pytorch and other things like that.
• 33:47
  And being able to wrap those up in functions and then organize
• 33:50
  those functions into a workflow as a sequence of steps, it's
• 33:54
  very elegant and useful. But the challenge with that is
• 34:00
  then that code is the only way you can understand what the word
• 34:04
  fill looks like, right? So if you want to draw a picture
• 34:06
  of it from your code, you have to run the code somehow and get
• 34:09
  an artifact. What is it?
• 34:11
  Artifact, right? You don't have ADSL, right?
• 34:14
  How do you draw, generate a diagram from that?
• 34:18
  Lot tools, lot of these things will do that for you.
• 34:21
  They will make a picture for you.
• 34:22
  And maybe you can save it as an SVG, maybe you can't.
• 34:25
  Maybe you can take a screenshot, whatever.
• 34:27
  But you have a picture, right? And you can get somebody that
• 34:29
  picture and they can say, this is what we're doing and you can
• 34:32
  have a discussion about it. And so I think the problem with
• 34:35
  the annotation side of things is then only people who can write
• 34:38
  code, you know, can't understand that workflow.
• 34:41
  And so that's I think is a challenge.
• 34:43
  And then I think it's feel like me and your ultimate goal is
• 34:47
  there's more workflows and there's nesting of workflows.
• 34:50
  There are people who don't write code.
• 34:52
  They're just talking about your machine learning workflow is a
• 34:54
  big, the part of a bigger system.
• 34:56
  And that's a big workflow. And we have pictures for all the
• 34:58
  rest of yours is a black box and we don't understand what it how
• 35:01
  it works and what the different failure states are and so forth.
• 35:05
  And so I think that it runs a follow of that.
• 35:07
  So I don't think that the the infrastructure as codes
• 35:10
  approach, the annotation approach is bad because it could
• 35:14
  produce artifacts that are the definition of the workflow.
• 35:18
  I just don't see a lot of evidence of that is where these
• 35:21
  tools are going right now. And maybe they just as they go
• 35:25
  on their adventure or building their systems and services that
• 35:29
  they'll something will come out right.
• 35:32
  There's a wide variance of what these DS LS look like.
• 35:36
  There's a lot of history there. There would have been some
• 35:38
  attempts to standardize it in various contexts.
• 35:41
  The science domain at a YAML based thing called Common
• 35:45
  workflow language BPM is something learned from the
• 35:48
  object management group, I believe as a standard for
• 35:51
  business process modeling. And they have a notation and
• 35:53
  that's another standard for the diagramming of them.
• 35:56
  But those, have they taken root in various communities?
• 36:00
  Sure. They're widespread.
• 36:02
  Probably not. And so I think the the it's not
• 36:05
  clear that there is a real winner there and it's not clear
• 36:08
  that we necessarily need a standard.
• 36:10
  But certainly within your organization, if you had 10
• 36:13
  different formats, you'd probably be unhappy.
• 36:15
  Yeah, You know. Keep going crazy, yeah?
• 36:19
  There's some work to be done there.
• 36:20
  But when it comes to these domain specific languages, is it
• 36:26
  something that you in an organization, you choose one and
• 36:30
  you go with it and you can abstract a way like you were
• 36:36
  doing at Stitch Fix. You mentioned you had the end
• 36:41
  users using the domain specific languages and then underneath
• 36:46
  the hood you had almost that infrastructure as code layer and
• 36:51
  it feels like that was working well for you all, yeah.
• 36:53
  Maybe the baby, depending on your person, like it was working
• 36:57
  well to keep the the system's longevity, right?
• 37:01
  So that as we change how those tasks were interpreted by the
• 37:06
  system, right, as technology changes the workflow is just
• 37:10
  metadata about what we would like to see happen and how they
• 37:13
  are chained together. So I think there's a value in
• 37:16
  that, but I think that there is definitely some push back that
• 37:20
  comes with that as well, because it's another thing, another
• 37:23
  artifact, It's external to your code.
• 37:25
  There can be mismatches. So there's lots of challenges
• 37:27
  there too. When and so I think it's really
• 37:30
  depends on the particular user that that you are interacting
• 37:35
  with. What's nice about most of these
• 37:37
  things, there's a high prevalence of people coding
• 37:40
  these things in YAML. It's the, you could like it or
• 37:44
  not. I, I don't find YAML a problem,
• 37:46
  but a lot of people don't like it.
• 37:48
  It's OK, but the structure's pretty much the same.
• 37:52
  So there's a list of steps and the step has a bunch of metadata
• 37:55
  and they all have names and they point to each other in some
• 37:57
  mechanism. And so the so to having that
• 38:01
  kind of common format lets you take a workflow from system A
• 38:06
  and a workflow from different system B.
• 38:08
  And if they're both in YAML, you can think about how you would
• 38:10
  represent those as they're artifacts that you can check
• 38:14
  into social control. You could generate diagrams from
• 38:17
  them. Maybe there's a tool and system
• 38:19
  for that. So I think there's having DSL
• 38:21
  has benefits like that where it's a, it's just something you
• 38:24
  can parse and you don't have to run code because that requires
• 38:30
  infrastructure that requires, you know, an environment set up.
• 38:33
  You can just parse it and understand what are the steps of
• 38:36
  that thing. And then people go, that's when
• 38:38
  people jump off and be like, oh, we should have a standard, but
• 38:41
  genders are hard, right? Getting people to agree.
• 38:44
  They take a very long time. I'm not saying that won't
• 38:46
  happen. I'm just it's not something I
• 38:48
  see happening right now. And but maybe there there'll be
• 38:52
  a need for it in the future, right?
• 38:55
  And then there's a whole custom side, right?
• 38:56
  That's another thing like people have created their own little
• 38:59
  languages, declarative languages for describing these things.
• 39:02
  That is definitely on the down side.
• 39:05
  I don't know that people are doing that so much anymore.
• 39:07
  It's like. That trend is fading.
• 39:11
  There's a reason why they do it. It's.
• 39:12
  You can make a nicer thing, right?
• 39:15
  But then you have the problem of to beep up to learn your syntax
• 39:18
  and semantics, and maybe that's more trouble than it's worth at
• 39:22
  this point especially. Onboarding new folks and
• 39:25
  everybody's got to go through that.
• 39:27
  And so you're now creating just a more cumbersome process to get
• 39:31
  someone up to speed. Yeah.
• 39:33
  And these things are, everybody likes to use the term DAG, which
• 39:36
  I don't always because not everything's A tag, right?
• 39:39
  The acyclic part, their workflows have loops, right?
• 39:42
  So they're graphs, right, in general.
• 39:44
  And sometimes they're forests. And since that there could be
• 39:47
  workflows that have two different independent pieces.
• 39:50
  There's lots of complex things out there.
• 39:51
  Those are the edge cases, right, Even things that have loops out
• 39:54
  the edge cases. So the DAG term is a
• 39:58
  simplification, computer science wise to make it nice to execute
• 40:03
  and but it's still, even with a DAG, you can have meets and
• 40:09
  joins, right? So you can have a little sort of
• 40:10
  loop in your thing and you got to cut that somewhere.
• 40:14
  If you're writing it in YAML, Jason, whatever, you've got to
• 40:17
  cut it and you've got to, you know, and how you make those
• 40:20
  choices. Maybe it's easier or less easy
• 40:22
  as the thing gets more complex and that's where you need tools
• 40:26
  in the end of the day. Or if you just have thousands of
• 40:29
  different Dags or workflows in the organization and it's, I've
• 40:34
  heard so many stories of folks who are like, yeah, we started
• 40:37
  as a start up and then we had success and the Airflow Dags
• 40:42
  just kept growing and nobody really went back and sorted
• 40:46
  those out. And so you have that sprawl.
• 40:49
  Yeah, I looked at when I said sys fix.
• 40:51
  We had to airflow underneath there.
• 40:54
  We had the system with the DSL, so I could pull all of the
• 40:58
  workflows through thousands that we had and look through them.
• 41:00
  And what was I found interesting is that there were some oddballs
• 41:05
  and they're like that did all kinds of crazy stuff.
• 41:07
  But most of them are a straight chain of steps, right?
• 41:11
  ABC chained together, right? That's most of what people are
• 41:15
  doing and it's not a surprise. So all this, it's a DAG, it has
• 41:20
  loops or it's not a DAG. Most people's like the 90% case
• 41:24
  is probably a, a straight through chain of of things.
• 41:27
  That's a bad numbers, I guess, but is what I, I found
• 41:31
  overwhelmingly the majority wave, higher than 50% of what
• 41:36
  and this was an organization. I've been doing this for a
• 41:38
  while, was these straight through chains.
• 41:40
  And it makes sense. There's some kind of preparation
• 41:43
  stages of what you're doing. And then there's the main event.
• 41:46
  You're training them all. You're doing inference, you're,
• 41:48
  you know, obserting into a database and then there's some
• 41:52
  clean up, maybe a band. And that's like most people's
• 41:55
  workflows. And that's, I think how these ML
• 41:59
  data engineering workflows differ from the business process
• 42:05
  workflows, right? Business process workflows
• 42:08
  aren't a straight through chain, right?
• 42:11
  There's decisions being made and about did we answer this
• 42:15
  customer's question, did that If that system fails, we go over
• 42:18
  here to do something else. If there's a transaction, when
• 42:22
  we've done the transaction goes through, the order gets made.
• 42:27
  If it doesn't go through, there's some other process,
• 42:29
  right? And so they think those business
• 42:31
  process workflows, they have much more complexity in them.
• 42:35
  They have a lot of branching and conditionals and they have a lot
• 42:38
  of side effects. Like if we succeed here, we're
• 42:41
  going to notify another system, we're going to send a message to
• 42:45
  the customer or something else like that.
• 42:47
  So they have all these side effects that happen along the
• 42:49
  way that are just not a thing as much.
• 42:53
  And although you can imagine them, they're just not in
• 42:56
  practice as much in MO OPS. And so that's how these systems
• 43:00
  are different and that's why they're different products for
• 43:02
  them. It makes sense.
• 43:05
  I I'm just thinking what you get with your favorite DAG tool is a
• 43:13
  slack message, right? In fact, like what are?
• 43:15
  Useful, right? Yeah, exactly.
• 43:18
  But it's not like what you're talking about with this super
• 43:20
  complex logic or loops happening or then spitting out into
• 43:25
  another subset of a workflow. Then it is fascinating to think
• 43:29
  about what I work like. I can't believe it took us
• 43:33
  whatever, 40 minutes, 45 minutes to get into this part of the
• 43:39
  workflow engines. But I have to ask about agents
• 43:45
  and what your take on them are as almost like workflow engines
• 43:52
  and workflows in general. And also I will preface this by
• 43:57
  saying we had Igor on here a few weeks ago and he was talking
• 44:04
  about how agents should be seen or even LLM calls should just be
• 44:09
  seen as another step in a DAG O. We're taking messy data and
• 44:14
  making it neat and tabular, and that's one of the steps of the
• 44:20
  DAG. But with agents, I've seen so
• 44:22
  many different examples of folks who have tried to explain the
• 44:29
  agents or have the agents work as Dags where they just make up
• 44:34
  the they make up the graph on the fly and you now are dealing
• 44:38
  with this workflow that was created by an agent.
• 44:43
  There are some companies out there in the mix that I looked
• 44:45
  at who are specifically focused on agentic systems.
• 44:52
  They, I put them in the category of sort of the business process
• 44:54
  because they're tools look exactly like that.
• 44:56
  Yeah. And they're a little more
• 44:57
  generic in the sense that they are.
• 45:00
  I would think of them supporting more like your chatbot type of
• 45:04
  interface where you're talking to your favorite airline and
• 45:07
  things are happening. And when you say, yeah, I'd like
• 45:09
  to buy that ticket or I have this problem with my luggage,
• 45:13
  you know, there's a, it interacts with a bunch of stuff
• 45:17
  and comes back to the agent interface, right?
• 45:20
  And that's a workflow that they have to manage and that's
• 45:23
  automated in some capacity. And so there's people who are
• 45:27
  building products for that kind of workflow.
• 45:29
  And I found them amongst the things that are with survey.
• 45:33
  I think that the sort of interesting side that's not that
• 45:38
  is this kind of challenge of the LM based serve agent system
• 45:43
  where you have some inference that's happening there and then
• 45:47
  there's a consequence. It's made, it's put something
• 45:49
  out there that was either code ish or it's something that's
• 45:54
  coming back to the user. And and that's part of this
• 45:56
  bigger workflow. And that's more like an embedded
• 45:59
  workflow. And that could be dynamic,
• 46:00
  right? As you say that it could be
• 46:01
  generated by the system itself. It could be generated by another
• 46:05
  piece of code or some other model.
• 46:07
  And there are these sort of part of the mix here are these things
• 46:11
  that are more like workflow engines.
• 46:13
  And they are so they're not systems necessarily as much as
• 46:17
  like it's a library to do this kind of workflow orchestration.
• 46:21
  And, and you can write your thing or have it give it a one
• 46:25
  of these dags of things to do and it will run it right and to
• 46:29
  some completion. And that's more like an embedded
• 46:33
  workflow engine. And there's a bunch of them like
• 46:34
  that out there. And that is interesting because
• 46:38
  it goes back decades to what people were doing with rules
• 46:41
  engines and for work Plus systems, because those were
• 46:44
  engines that they put in, you put inside a product.
• 46:46
  And sometimes they were like desktop applications that were
• 46:48
  doing this stuff and they're running the rules and acting on
• 46:51
  your behalf or, and there's a user in front of them.
• 46:53
  So now we've got like a chat bot interface or agent out there
• 46:56
  that somebody's interacted with on our website or through an
• 46:59
  app. And it's doing the same thing,
• 47:02
  right, but different scale, right?
• 47:04
  And it's not running on your desktop app, it's running out
• 47:07
  there in the cloud somewhere, but it's the same thing.
• 47:09
  It's an embedded engine running an embedded workflow and dynamic
• 47:13
  or not. And so I think that's what some
• 47:15
  really that's there's just some cool possibilities there.
• 47:19
  I have not. And this is a good research
• 47:22
  hopper for somebody or myself. I just, you know, where are you
• 47:25
  in that, you know, that adventure, like what is what
• 47:27
  have people done successfully? How is it architecturally
• 47:31
  different from what people are doing now with these interfaces?
• 47:35
  Where's the sort of there there and what can't you do with the
• 47:39
  current systems you could do with this sort of theatrical
• 47:42
  embedded LLM based thing, Our agent, there's a lot of
• 47:47
  possibilities. What I hadn't thought about
• 47:49
  before that you just opened my mind to is how the agent is
• 47:54
  almost the gateway to choosing the right workflow.
• 47:58
  So we talked a lot about agents being able to choose tools or
• 48:02
  have access to tools. And most people think, OK, now
• 48:06
  it can scrape the web or it can have access to my database.
• 48:09
  But the thing that you just said is, yeah, one of the tools might
• 48:14
  be that it kicks off a workflow. And so then you don't have to
• 48:20
  worry about the agent spawning a workflow every time.
• 48:24
  And then maybe it spawns the wrong workflow or the workflow
• 48:28
  isn't exactly like what you need to happen.
• 48:31
  So the agent just has to choose between what workflow it needs
• 48:36
  to use. And that is very much like going
• 48:41
  back 20 years, but now we have a little bit looser way of having
• 48:46
  the end user interact with the agents or the if that then this
• 48:52
  statement. Yeah.
• 48:53
  With the, I mean that's kind of the interesting juncture that
• 48:57
  we're at where a lot of the, you know, large language models and
• 49:00
  we have regions systems and so we've taken them apart workflow
• 49:04
  systems. So we know we have these kind of
• 49:06
  pieces that are much more advanced than they were back.
• 49:09
  And then they're not all in one system, just doing pointed in
• 49:13
  One Direction. And so we can take these puzzle
• 49:15
  pieces, put them together, they're like Legos and make
• 49:18
  different things out of them. And, and since the technology is
• 49:21
  more advanced, we can do sort of really amazing stuff for them.
• 49:24
  And I think that's the nice juncture where we're at.
• 49:28
  I was surprised going through the list of projects, how many
• 49:34
  there were that were still active in, in all of the
• 49:38
  categories that I have. And that's 79th scientific
• 49:41
  number. There's a lot of noise out there
• 49:43
  and but there's a lot of just people who are actively
• 49:46
  developing these things and maintain them.
• 49:48
  They're using them for things and they're in a variety of
• 49:51
  contexts. And so I think that maybe not
• 49:54
  what you hear out there right now in terms of the buzz in the
• 49:58
  industry that this is a very healthy, vibrant area of work
• 50:02
  that people are doing. They're using it for stuff,
• 50:04
  obviously, because they're, these projects are active and
• 50:08
  some of them are niche and at the corner and some of them are
• 50:10
  commercial products and, and that people are are selling and
• 50:14
  everything. And so I think that's good for
• 50:17
  users out there because they can find the thing that matches it.
• 50:20
  The only challenge is that there's a little bit of tyranny
• 50:22
  of choice, right? Like.
• 50:25
  If you're new to this and you're like I need a workflow system
• 50:28
  for X, you've got some choices that you're apartment to make.
• 50:31
  How? Do you categorize like the RPA
• 50:35
  systems, would that be the business ones that you're
• 50:38
  talking about? And RPA, what do they call it?
• 50:41
  Robotic Processing Automation I think is what it stands for.
• 50:46
  I don't tell that, so I'm not trying to think of.
• 50:48
  Did I run into anything that was more on the automation
• 50:52
  manufacturing side? And I'm going to say that I
• 50:55
  didn't see a lot there and what I surveyed.
• 50:57
  And so that might be a whole different thread of this.
• 50:59
  Certainly there are lots of people doing things like we did
• 51:03
  at my last company where we're sending a protocol to a like a
• 51:08
  liquid handling robot because it's part of our automation of
• 51:11
  what's happening in allow, right?
• 51:12
  And, and that kind of like in biotech and in general like
• 51:17
  pharma, the kind of automation you can use these workflow
• 51:21
  systems that are, they've been talking about here because it's
• 51:24
  a service call to something and their thing has an API and you
• 51:27
  push a protocol to it and tell it to go.
• 51:30
  And so that kind of level of automation, I think people are
• 51:32
  using these kinds of tools for. But the sort of more industrial
• 51:36
  things, there's, there's a whole other world there.
• 51:39
  I learned that in science, there's a whole other world of
• 51:43
  plant automation that uses a different technology and it used
• 51:49
  some really old technology, which is scary.
• 51:51
  Like it uses OPC, which is Microsoft Ole from like 1995.
• 51:59
  What? And, and so, you know, that's
• 52:02
  why you can hack these like a power plant.
• 52:05
  For example. Yeah.
• 52:08
  Hold it for ransom. Those are so there's some areas
• 52:11
  there that there are completely like off the radar from this
• 52:14
  kind of group of workflow systems and there might be
• 52:17
  really good tech there. I just, I think that's a whole,
• 52:20
  there's a whole another world. Yeah.
• 52:23
  So you've been seeing the trends of what's dying and what's
• 52:28
  growing and you did all this research on it.
• 52:31
  Do you feel like you have any bets or guesses on what the
• 52:37
  future of these systems holds? I I think there are some good
• 52:43
  contenders in the SAS realm for machine learning workflows in
• 52:49
  general unlocks and they have some nice tools.
• 52:52
  I think the challenge for machine learning AI context is
• 52:56
  that we're just getting started. And so their customer base are
• 53:00
  all these sort of early adopters and there are startups and
• 53:03
  people like that. So if you take this thing and
• 53:05
  you walk into a large enterprise where that's regulated and that
• 53:11
  is, you know, has like air gap systems, right?
• 53:16
  They can't use a SAS system. Yeah.
• 53:18
  And there's big applications in these places making that leap
• 53:22
  from you can use our SAS service and everything's cool and we're
• 53:25
  writing Python code and we're doing all this cool stuff with
• 53:27
  it to like these places where it's highly regulated.
• 53:31
  It's and there's all these other enterprise challenges and you
• 53:34
  have to have like certain kinds of certification to be able to
• 53:36
  operate in there. That's where the revenue is for
• 53:40
  these companies to be potentially to go to.
• 53:43
  And so there's just sort of that they have to make that leap and
• 53:45
  then there are some that are doing that.
• 53:47
  So I think the the evolution of some of these companies to be
• 53:50
  able to provide Enterprise Products that really meet the
• 53:54
  needs these other needs I've asked to have certain security
• 53:58
  levels and certain compliance things and be able to work in
• 54:01
  these sort of non cloud based environments and so forth or
• 54:07
  private clouds and so forth. That's but that that kind of is
• 54:11
  a growth area, right? And you have to be able to
• 54:13
  survive that because it's also expensive.
• 54:15
  So that's I'm watching to see who matures in that realm.
• 54:19
  That's why some of the projects that are open source and good
• 54:22
  technology and well supported work well in these places
• 54:25
  because that their technology teams can take them and put them
• 54:27
  inside these environments and deploy them, but then they have
• 54:31
  to manage them sex. So I think that's a trend.
• 54:33
  I was happy to see that. I think the business process
• 54:36
  automation thing, it seems like there's a health community of
• 54:38
  people using that. I think the agent tech systems
• 54:43
  is a area of growth for them to some extent.
• 54:47
  The MLTI stuff is an area of growth for them.
• 54:50
  I'll be curious to know if they get traction in there versus
• 54:52
  just saying we do MLA. If are people turning to these
• 54:56
  other products that are a little bit older, we'll see, right?
• 55:01
  It it's almost like with the business process automation I
• 55:04
  see now there incorporating the capabilities of LLMS into their
• 55:15
  products. And so now as one of their steps
• 55:19
  in your whole workflow, you can add whatever an LLM call is
• 55:24
  capable of doing, whether that is summarizing a bulk of text or
• 55:30
  it is going in, you scrape a website.
• 55:34
  And then from that scraped website, you pick out the most
• 55:37
  important stuff. And now you have that data to
• 55:41
  pass on for the next step. And so you've got, you've got
• 55:46
  new tools that you can work with.
• 55:48
  And I've seen it done really well with friends and marketing
• 55:52
  who are trying to create content.
• 55:56
  And what they do is because you've got the almost like the
• 55:59
  easy way of creating AI slop, which is just saying to ChatGPT,
• 56:04
  create me a blog post about whatever GPU consumption in the
• 56:09
  US and then it'll spit out whatever it has inside of it.
• 56:13
  Or you can start with saying create me an outline, find like
• 56:18
  3 relevant blogs that talk about it and then you choose as a
• 56:23
  human different blogs that you like.
• 56:25
  And then it it uses the information from those blogs, it
• 56:29
  creates an outline for it. And then you say now create the
• 56:32
  intro paragraph. Now create 3 body paragraphs and
• 56:35
  you really prompt engineer it to be a much more in depth type of
• 56:40
  workflow. And the tools are now giving you
• 56:45
  those capabilities by default because you have the LLM calls.
• 56:50
  I think when interesting possible evolution here would be
• 56:54
  that you have engineering teams, which are often very expensive,
• 56:59
  taking things like gender of AI technology in general, LMS and
• 57:04
  and they're doing various things to in their quality checks and
• 57:07
  so like that does this thing pass our tests?
• 57:10
  Is it they risk assessment test to make sure that it's not going
• 57:13
  to do something bad, spit off bad results and their guardrails
• 57:16
  and all this stuff. And so you imagine that there's
• 57:19
  these more technical workflows that they can take a new model,
• 57:23
  run through its paces and say this is good, this is not good.
• 57:27
  Or there's some score maybe how it's not a black and white type
• 57:31
  of thing, it's a score of how well does it do in these
• 57:34
  different dimensions. But as there's more to do and
• 57:39
  there's more uses of these technologies, you can imagine
• 57:42
  that there is a higher level user there that's not on the
• 57:45
  technical team who this is their product and there's a new model
• 57:50
  and they or there's a fix or there's a better model where it
• 57:53
  has it doesn't do the bad thing because they they tested the new
• 57:56
  version of it and there's a workflow for accepting that and
• 58:01
  rolling it out to their organization or in their
• 58:03
  application. If that always has to be a
• 58:07
  technical engineering problem, that's expensive.
• 58:10
  And so I can imagine that part of the way that people build,
• 58:14
  use workflow systems is that kind of multi layered thing
• 58:17
  where there's technical workflows, they use a very
• 58:20
  specific technology that's geared towards the task at hand,
• 58:23
  training a model, evaluating it, producing these risk scores.
• 58:27
  And then there's a the business level workflow.
• 58:29
  It's saying, how do I take that model and get it into my
• 58:32
  application, roll out to my users, get into production and
• 58:36
  there's a gatekeeper which is theirs.
• 58:37
  And this is where human in the loop, which we haven't talked
• 58:39
  about, but there's a human in the loop step there where it
• 58:43
  says, do I want to do this? Because there's a, there's a
• 58:46
  business decision to, to roll this thing out.
• 58:49
  And when you codify that as a, as a workflow and that one way
• 58:53
  it gets out the door is that somebody goes and does that
• 58:56
  human in the loop step of saying yes, as a human decision maker.
• 59:01
  It's not just a technical team somewhere who, you know, if they
• 59:06
  do the right thing in their DevOps, you know, thing, it goes
• 59:09
  out the door. It's a, you know, maybe they'll
• 59:12
  even know they no longer have that ability.
• 59:14
  It's only done through the workflow system and there's a
• 59:16
  human who makes that decision and it's traceable in your
• 59:19
  organization and it maybe is less prone to mistakes,
• 59:23
  accidentally rolling out the wrong model or a model that
• 59:26
  doesn't pass your tests. And so that kind of like level
• 59:30
  of control and then bringing it out of the engineer organization
• 59:33
  and back into the hands of set of a park manager or business
• 59:36
  user of some sort, I think that's going to make the thing
• 59:39
  cost less. You're going to get better
• 59:40
  results in terms of quality. That's where a workflow system
• 59:44
  of different and different layers of them can be super
• 59:46
  helpful. I would love to see that kind of
• 59:48
  like thing, these kind of things already exist, but they're run
• 59:52
  by technical teams, right? They're using the same tool to
• 59:56
  do their DevOps, their operations, right?
• 59:59
  And so the tools are all there. But I think that business
• 1:00:04
  process automation side of it is not as much because again, those
• 1:00:08
  people are usually like, yes, engineering team roll out the
• 1:00:11
  new version that's a Lac message to a person and then a person
• 1:00:15
  goes and does a task. And looks good to me, yeah.
• 1:00:19
  And that's, that's where we are. I think the challenge with LMS
• 1:00:22
  is there's they're squishy, right?
• 1:00:24
  And somebody needs to look at these risk scores and there's
• 1:00:29
  new benchmarks and cool things coming out to not say.
• 1:00:32
  And then they make a decision about is this risk receptor,
• 1:00:35
  right? Roll this new version of
• 1:00:36
  whatever model we're using from whatever provider.
• 1:00:39
  And we've done our tests, we have our evaluations.
• 1:00:43
  And now I have to make a decision in my job as the
• 1:00:45
  product manager as though whomever just to roll this thing
• 1:00:49
  out. And you don't want to skip that
• 1:00:52
  and you want to record it and you want to just so that we
• 1:00:55
  understand how did this thing get out?
• 1:00:57
  And then want to change your process or change that workflow
• 1:01:00
  to to deal with whatever issues your organization might have in
• 1:01:05
  terms of the their use of these AI technologies, right?
• 1:01:09
  Yeah, it's another. Garden, but it's a like a
• 1:01:11
  business card. 100% in the fintech or just financial or
• 1:01:18
  even just any regulated space that is a necessity because
• 1:01:23
  they're going to get audited. And so you got to have that
• 1:01:25
  explain ability of what exactly you were thinking when you put
• 1:01:30
  that out there. And so it makes sense that this
• 1:01:32
  would only come in a workflow so that you have that specific area
• 1:01:38
  where someone pushed the button and said, yeah, we're good with
• 1:01:41
  that and the logic behind why they chose that.
• 1:01:45
  But I, I do like the idea of taking it out of engineering's
• 1:01:51
  hands and just getting a different set of eyes on it
• 1:01:53
  because of the, as we had, we had Allegro on here a few weeks
• 1:01:58
  ago and she was talking about how she's a big proponent of
• 1:02:02
  density of diversity in a room. And so by doing that, by taking
• 1:02:08
  things out of engineering's hand, then it's not only
• 1:02:12
  engineers that are looking at it.
• 1:02:13
  And so by way of that you have higher density of diversity.
• 1:02:18
  Yeah, that sounds great. And Amorphosisms are a good tool
• 1:02:22
  for defining that process and recording it and frame metadata
• 1:02:27
  and collecting information along the way.
• 1:02:30
  So you have a trail of information of what you did and,
• 1:02:34
  and, and then you can act on that trail in terms of making
• 1:02:39
  your processes better or when just somebody wants to know my
• 1:02:42
  versions of this model, are we using outer assistance?
• 1:02:46
  They might be different right now you have that trim, right?
• 1:02:49
  And people are doing this again, some other way.
• 1:02:53
  They maybe they have systems for this, but but maybe they're not
• 1:02:56
  using these tools that are available and they not have to
• 1:02:59
  build a smash hole system for it.
• 1:03:00
  They can just use a workload system that exists out there.
• 1:03:04
  And that's so that there's a that sort of build by choice
• 1:03:09
  then too, right? And you know, what this also
• 1:03:11
  makes me think of is how a friend told me about how wild
• 1:03:17
  any enterprise is right now when it comes to what AI they're
• 1:03:23
  using. And not just like your ML teams,
• 1:03:28
  but if you think about governance in the enterprise
• 1:03:32
  level or if on the organizational level and you
• 1:03:34
  have maybe the marketing team is using one business processing
• 1:03:39
  software that has some LLM calls or capabilities, AI capabilities
• 1:03:44
  within the tool. Then you've got the actual
• 1:03:47
  software engineers that are using these AI coding helpers
• 1:03:51
  and you've got the HR team that's using some SAS software
• 1:03:55
  that has some kind of AI capabilities.
• 1:03:59
  And you don't realize it, but you as an organization, you are
• 1:04:04
  exposed. If your idea is like, yeah,
• 1:04:07
  we're keeping all of our data inside, that is totally out the
• 1:04:10
  window because everybody's using a different piece of SAS tooling
• 1:04:15
  that potentially is sending the data wherever it needs to go.
• 1:04:20
  And so from a governance perspective, bringing these
• 1:04:23
  workflows in and recognizing that if it's workflow, if you
• 1:04:28
  have it documented in workflows, you also have a bit tighter
• 1:04:33
  control on what is being used and how it's being used and how
• 1:04:38
  these things are happening, I would assume.
• 1:04:41
  But I can imagine that it also can slip through the cracks
• 1:04:44
  there too. Yeah, yeah, for sure.
• 1:04:46
  I mean that there's a lot of data governance challenges in
• 1:04:49
  the world here today that that are are made worse.
• 1:04:56
  And I think, and I'm not sure there's a clear way out of that
• 1:05:00
  box right now, even with workflow systems, but.
• 1:05:03
  Yeah, it's not going to, it's not going to help that much.
• 1:05:06
  That's the truth, man. But just the data governance
• 1:05:11
  piece, I know I had a friend tell me that their company did
• 1:05:16
  an audit and they were expecting that folks were going to be
• 1:05:20
  using like 10 tools, the AI tools.
• 1:05:25
  And after they did the whole audit and this is a come, this
• 1:05:27
  is a relatively small start up, like 200 folks mid size is what
• 1:05:33
  you would call it. What they found is that there
• 1:05:36
  were over 92 tools that were being used that had AI
• 1:05:41
  capabilities or were AI tools. And they just were looking at
• 1:05:47
  each other thinking like, wow, this is wild.
• 1:05:50
  And a lot, there was a lot of repeat tools.
• 1:05:53
  So you have a lot of the same workflows, but maybe it's
• 1:05:56
  different parts of the organization's different
• 1:05:59
  branches. Maybe you're paying for, even if
• 1:06:02
  you are OK doing the Open AI Enterprise edition, maybe 1
• 1:06:08
  branch is paying for it, another branch is paying for individuals
• 1:06:13
  to be using it. And so all that governance is a
• 1:06:16
  mess. So that was the challenge in my
• 1:06:19
  last company because partially because I am not a a molecular
• 1:06:23
  biologist. That's right.
• 1:06:24
  And so I don't know what is standard tooling and things and
• 1:06:30
  resources that they use online. And, and the way that we took
• 1:06:34
  that apart was this process engineering aspect was having
• 1:06:38
  those discussions and drawing those pictures of what is, what
• 1:06:43
  do you do on your day to do this task and how, what systems and
• 1:06:47
  like, where do you get that result from?
• 1:06:49
  Oh, we go online and we take that DNA and we stick into this
• 1:06:53
  tool and we run a blast search and and then we do this other
• 1:06:56
  thing with this other site because they're good at this
• 1:06:59
  particular thing. I'm really interested and you're
• 1:07:00
  like, OK, and there's so there's all these like touch points to
• 1:07:03
  different systems and these were things that are very unique,
• 1:07:08
  isolate strains, stuff like that.
• 1:07:09
  So that data, the genome is our was our sort of bread and
• 1:07:13
  butter. And you have this sort of
• 1:07:15
  challenge of if we take a little snippet of it, nobody knows
• 1:07:18
  where that came from. And so we're OK with that going
• 1:07:20
  out. But there's a, there's a fine
• 1:07:22
  line, maybe a green line there of like how much is too much?
• 1:07:26
  Just knowing those interactions. It all comes from doing this
• 1:07:29
  sort of I, I think is the process there.
• 1:07:32
  If you can draw a picture like that, flow chart, whatever, pick
• 1:07:35
  your favorite tool and then OK, what am I talking to?
• 1:07:40
  It also helps you with the data artifacts problem, which is,
• 1:07:42
  yeah. So what data went in?
• 1:07:44
  What data comes out? Do we care?
• 1:07:45
  Do we store that? Where does it go?
• 1:07:47
  Does it go into some knowledge graph?
• 1:07:49
  Does it need to be stored for to have a sort of full view of the
• 1:07:53
  experiment or whatever we're doing here for your compliance,
• 1:07:56
  the industry, like maybe you need to record that because it's
• 1:07:59
  a, you're required to record those interactions that like is
• 1:08:05
  essential to your business. People do this, but they don't
• 1:08:08
  necessarily do it in a uniform way.
• 1:08:10
  And so that's where these like I, I was pushing on this using
• 1:08:14
  this thing called BP and men. This was something, it just was
• 1:08:17
  a nice notation. Like it's a visual notation that
• 1:08:21
  has people have considered all these problems.
• 1:08:22
  And so we don't have to make one off.
• 1:08:24
  We can just use it. There's tools for and, but that
• 1:08:27
  also means you can also say, OK, I'm going and talking to the SAS
• 1:08:30
  service. What does that service do?
• 1:08:31
  And then you can, you know, decide whether you trust.
• 1:08:35
  Them. What data are you giving it?
• 1:08:36
  What are you? What's your end goal?
• 1:08:39
  Yeah. How are you trying to use it?
• 1:08:41
  Because maybe we already have another service that we're
• 1:08:44
  paying for and you're paying extra for that.
• 1:08:46
  I know that we had Maria on here probably two years ago now and
• 1:08:52
  she did this with her company a Hoyts because they did not have
• 1:08:57
  any centralized ML platform. And so they just went to all the
• 1:09:02
  different teams and said what are you using?
• 1:09:05
  How are you using it? And they recognize that they
• 1:09:09
  already had a lot of usage on data bricks.
• 1:09:13
  So they were like, I think we should probably standardized the
• 1:09:15
  data bricks so that it is cleaner and just recognizing
• 1:09:20
  from talking to people and drawing it out.
• 1:09:23
  I really think there's so much value in that, but what you're
• 1:09:27
  saying is even more. It's like you're taking an
• 1:09:30
  etching sketch and you're making a beautiful picture of what is
• 1:09:35
  happening at each step and what the goal is for each of these
• 1:09:41
  steps and being able to have that.
• 1:09:43
  You then are so far ahead of the curve because you understand
• 1:09:50
  each person's tasks, what they're doing, how they're doing
• 1:09:54
  it, but you also understand how that fits into different
• 1:09:58
  workflows, as you were saying, and and not looking at it as one
• 1:10:01
  big workflow, but seeing all the different workflows and how they
• 1:10:05
  interact with each other. Because then you can ask all
• 1:10:08
  kinds of great questions. Or is it worth automating this?
• 1:10:11
  You know, is it and do we need to be worried about this data
• 1:10:14
  leakage from the service that you're using?
• 1:10:16
  Is there a better provider for that?
• 1:10:19
  If we were to use some kind of new machine learning tech and
• 1:10:23
  generative AI or an LLM or whatever, some model, where is
• 1:10:28
  it provide the most value in this process that we have this
• 1:10:32
  big and, and you can ask all these great questions and you
• 1:10:35
  can see what you know and you can get samples of your data
• 1:10:37
  because now you've done that analysis, It might seem old
• 1:10:40
  school and hard to do, which is the challenge.
• 1:10:42
  I have this challenge. This seems like a massive
• 1:10:45
  undertaking. Let's start simple.
• 1:10:46
  Let's start like a little piece of it or maybe a really big
• 1:10:50
  block diagram with like big chunks of what we do on a daily
• 1:10:53
  basis. And then there's they're just
• 1:10:55
  black boxes and what you need to dig into that when the time is
• 1:10:58
  right. But I think those kind of like
• 1:11:00
  different ways of, of making the problem smaller and more useful.
• 1:11:05
  You can be very strategic and where you start, but you get
• 1:11:08
  these sort of, and I think that's just a, it's probably a
• 1:11:10
  bigger challenge for bigger organizations.
• 1:11:12
  But I also think that smaller companies you need to think
• 1:11:15
  about this because if you're going to use machine learning
• 1:11:19
  tech and in general, but there's just inference or if it's like
• 1:11:24
  I'm building models like that, it's not just those MO
• 1:11:28
  engineering teams in a corner that need the workflow systems
• 1:11:32
  and the process engineering. It's everything around it as
• 1:11:36
  well. And so you can, you can go
• 1:11:38
  bottom up, but that's a hard sell.
• 1:11:40
  You can go to more top down in terms of like where is this
• 1:11:44
  going to provide value and organization and what do we in
• 1:11:46
  our industry, whatever it is, need to be worried about.
• 1:11:50
  And you have that again, like a picture is an amazing thing and
• 1:11:54
  ways to communicate outside of those technical groups that are
• 1:11:58
  used to graphs, right? So.
• 1:12:02
  You know what I find the most difficult in these is when you
• 1:12:06
  update processes and updating them on all the documentation
• 1:12:13
  and then making sure that, OK, this is the newest way that
• 1:12:17
  we're doing things. Even though maybe it's an
• 1:12:19
  experimentation for a few weeks before you really realize if it
• 1:12:23
  is a better way of doing it. And most of the time I'll update
• 1:12:30
  a process and it's on the fly. And then later maybe you don't
• 1:12:34
  codify that as well. And so it's updated in one place
• 1:12:38
  but not in another. And so that just is that the
• 1:12:41
  entropy of all of the stuff that's happening is really hard
• 1:12:46
  to wrangle. And you get it.
• 1:12:47
  It's like a workflow debt, I guess you could call it.
• 1:12:51
  That, and that's where, if the workflow system is how you get
• 1:12:54
  things done and, and it can produce documentation and, and
• 1:12:59
  diagrams and things like that. And, and that's where you go to
• 1:13:01
  look up like how are we actually doing this?
• 1:13:03
  That's one of the best sort of core benefits is, is you, if
• 1:13:08
  there's no out of sync, because it's how you do things and the
• 1:13:12
  challenge when it fits into a bigger process that isn't
• 1:13:15
  automated, but I think it's a useful starting point to have
• 1:13:18
  those discussions. And I think it makes what your
• 1:13:22
  every the technical task easier to do when even if it's a
• 1:13:25
  documentation piece artifact that you, you're drawing this
• 1:13:28
  diagram that you go back to the diagram and say, so explain your
• 1:13:32
  change that you want in terms of changing this diagram, right?
• 1:13:36
  And then you've started with the change to your description, your
• 1:13:40
  architecture or whatever, and the process, the workflow, and
• 1:13:44
  then you go off and build the thing.
• 1:13:47
  And then there's a kind of a reconciliation sometimes of, you
• 1:13:51
  know, the reality that I found is I went to go build it.
• 1:13:54
  That's maybe not quite matching the what we thought that they'd
• 1:13:56
  be any, but that's part of the sort of doing an agile process.
• 1:13:59
  You should be coming back to that thing and revisiting and
• 1:14:02
  having a little bit of structure there and diligence helps.
• 1:14:06
  And, but you don't have to, doesn't have to be overwhelming.
• 1:14:08
  And but if everything was workflow, it would just all be
• 1:14:12
  up to date, right? That's an ideal version of this
• 1:14:15
  thing. It's not a, it's not a reality.
• 1:14:17
  Of course it would be so much easier.