Latest / The Data Splash / How Atlassian Is Automating Data Engineering
Transcript
- Ido Bronstein: Welcome everyone to the Data Splash podcast brought to you by Upriver. If you are a data engineer, a data leader, or just someone trying to make sense of what the hell is going on with AI and data right now, you are in the right place. Here with me today is Prakash Reddy, head of data engineering and AI enablement at Atlassian. And we will talk about how he automate the data engineering work. Prakash started his career in big data when big meant Oracle. If you enjoy this, follow the Data Splash podcast. We have got a lot more coming your way. ⁓ Until next time, keep your data flowing and we will see you next time in the stream. pretty long time ago, and he was one of the early data practitioners at LinkedIn, leading data products for ads, and he also an investor and advisor for a lot of startups. So now, Prakash say hi to everybody. Prakash: Hey everyone, nice to be here. Thanks, Ido for having me. I'm really looking forward to this. Ido Bronstein: Yes, we're also excited and before we are diving ⁓ in we have a small tradition. This is the 30 second splash. So no time to think, just instinct. So let's jump in. Prakash: Okay. Ido Bronstein: Snowflake or Databricks Prakash: Databricks. Ido Bronstein: most underrated skill in data. Prakash: I would say data modeling. Ido Bronstein: Interesting. Dream vacation destination. Prakash: I would say any place which has a cabin, a lake and a mountain would be a dream vacation for me. Ido Bronstein: Is AI going to replace data engineering? Okay, we will arrive it in a minute. Prakash: You're getting in trouble, but my answer would be probably no. Ido Bronstein: No. No. Yeah. No. And it's going to be interesting and here where it will help. So to give our audience a short background, let us hear from you a bit, about Atlassian, ⁓ data organization ⁓ how guys structure, ⁓ which ⁓ you are getting from your data today. Prakash: Yeah, awesome. I'm assuming that most of the folks of your audience already know what Atlassian does. ⁓ For folks who don't know, we are the company behind Jira Conference, one of our flagship top products that almost everyone knows or have used in the past. ⁓ So data engineering for Atlassian is pretty huge. We are a sizable organization. ⁓ So from our, the way we support our company and our products is we have data teams that work with our ⁓ business functions, let's say sales, marketing, customer support, finance, workplace, et cetera. Then we have some data engineering teams that actually work on our products, ⁓ like Jira Confluence, et cetera. And we have some data teams that actually ⁓ are primarily more horizontal in nature where they cut across these business lines and products and build, let's say, our metrics, which we furnish to Wall Street, et cetera. These are three big pillars of data teams that we have. Now each teams have a certain specific focus of their own. Like I said, the business driven data teams, they are mostly geared or aligned towards making the business team successful, providing the data analytics capabilities, ensuring that teams can make decisions at warp speed, right? So that is the goal of those data teams. Then the product data teams who support products, they typically build your data pipelines to... understand how the products gets used by Atlassian's customers. We run experimentation pipelines to support our A-B tests that our engineers do and data scientists do. A lot of product feature usage analysis of what product features customers use, what don't use, what's the journey like, what does a sessionization look like of people using Atlassian products, how do they navigate between a product A to product B, and just understanding the behavioral aspects of what helps customers to bring ⁓ productivity in their company using Atlassian Product. So we try to use that understanding to kind of ship better products or improve our existing products. And then the third pillar, I said, ⁓ cutting across all of these things, building our top level metrics. And then we have these data science organization teams who support, work with data engineering teams to build insights, analytics. We have research teams that do research on customer product. So we are an ecosystem of data organization who work with our engineering business teams. Ido Bronstein: or Prakash: and ensure that Atlassian is set up for success. Ido Bronstein: So in short, you have everything AI, BI, external, internal, high level dashboards, day to day operations, basically everything. Prakash: It's a pretty big gig, would say. engineering is really big for Atlassian. We call ourselves data-driven. We make every decision about our product, about our business using the data. So data engineers, job is very safe, I would say. Ido Bronstein: Haha. So you and I had a chance to talk a couple of months ago and you shared with me the idea of using AI in order to completely change how you do data engineering in Atlassian. So first of all, I would love to hear a bit about like how you frame it, how you think about it, how you arrived to this idea. Prakash: Yeah, that was a fun meeting. I know that you and I met in Dallas. It was fun to kind of connect with you, learn about what you folks are doing. I think so far, whatever I've learned about you folks, think you guys are doing some amazing autonomous data engineering automation. Fun to see what you guys are doing. Now coming to Atlassian, ⁓ I think about six months ago or so, ⁓ we had this vision in mind that a lot of what we do as data engineers today are repeatable. they are automatable and we have automated some parts for data learning workflows today, which is normal scripting and ⁓ programming, right? But the potential of AI ⁓ was realizable for us when we actually migrated from our old data lake to the new data lake. We went through a multi-year migration where we ⁓ didn't have a... ⁓ environment isolation of our data lake back in the day where production development staging was all in one. We didn't have any repeatable patterns of how data engineers would develop ETL pipelines. Everybody had their own way they would do it. And we used to have that as a known problem back in the day. But with our migration to the new data lake, we actually standardized our patterns of how data products get built. We started to organize our data products better in terms of bronze, silver, gold, making sure we have environment isolation of production, development, staging, and all of that stuff. And we made a big shift to go from do it your way ETL pipelines to a declarative style ETL pipelines. So now there is a more uniform way of building data products that we do across all of our data engineering teams. Now, as we came to the declarative style of data pipelines, it gave us this unique opportunity. Now that we know there are very few patterns of building data products. Now you can unleash the power of AI to say, okay, now we have single pattern. Everything is codified, YAMLified. Let's bring AI automation on top, right? So I remember four or five months ago, I wrote this one pager. I named it as AI4D Vision document where I did some industry research of where industry is heading towards. How good is AI or LLMs on understanding data type workloads? And I put together a... vision doc of what are the key opportunities we have on the table across the SDLC process for data engineering. We broke it down into several sub capabilities of the SDLC of data engineering and we started to see, is there a true ⁓ area where we can focus on to get the biggest ROI to use LLMs and AI to automate some things and make it autonomous as we kind of mature our way. The biggest motivation I don't like you asked me was, Data engineers are always underfunded. We always are behind where we have so much work to do to help our company build more assets, drive analytics. We are always short of people. And the only way we could scale and build more and catch up with the company speed was if we could automate some things using AI and use AI as a force multiplier for data engineers day-to-day workflow. And that was our biggest motivation to kind of get into this. automating and detaching workflows. Ido Bronstein: Amazing. So a couple of things that I heard. First of all, that before you're jumping to AI, you need to validate that you have good foundations in your architecture, in your data. And only after you build everything in templates and structural way, you can trust AI that now he will help you to scale this architecture. And second thing, and I think it's very important, it's very nice to hear that. The need was obvious like data engineers, your team was always under pressures, needed to deliver more. And in way, AI is the solution that is very easy to sell to Atlassian because you always in a shortcut, as you said. Prakash: Let's go. Ido Bronstein: Interesting. And when you looked on the different stages of the workflow, what are those stages and where did you saw the biggest ROI? Prakash: Yeah, so ⁓ throughout the SDLC process of data engineering, ⁓ we outlined there is initial phase where we do requirements gathering. We want to understand what business wants us to do, a good understanding of what our current data model is, depending on the business needs, what all changes do we need to make? Is this a new build out that we're doing for data, new data modeling build out we are doing or we are improving our existing ones, enhancing our existing data models. So I think that's the first stage of requirement gathering. thought maybe we can spend some time here, but is this really going to give us the biggest ROI? We probably thought not really. So we did not spend much of our energies and efforts there, but wherever we knew that their existing data model is already there, we use dbt, airflow, data breaks, all of that technology stack as we have today here. We knew that for existing tasks of, say adding new columns, modifying, let's say existing business logic, making incremental changes that we kind of get a lot of these requests for. It's almost like we get inundated by these asks to update an existing data model to evolve because of some business evolution has happened. And this was already happening by a thousand cuts across DE teams. We felt like if we could take these small, small tickets that we get from our business stakeholders to improve our data model, can we automate that first? Can we get help from AI to make the code changes for us? Can we build the right skills to understand the business ask from a JIRA ticket and kind of deploy an AI agent with the right skill to go and tackle that problem and create a pull request for us. And the data engineers can either review the pull request, give it more information and can kind of iterate on that, right? So we wanted to tackle that as a first problem. Second area where we felt we spent a lot of our time is doing data ops, managing on-call, data failures, production support. And we definitely saw that there was a good amount of ROI we could get if we could build a on-call agent. that can look at a job failure, a ⁓ question coming from our stakeholders on data quality issues, investigate it for us as to what is going wrong and give us some proactive recommendations to the on-call engineer saying, hey, I've found out all these things. These are my recommendations to go and fix. Do you want me to do it or do you want to do yourself? We wanted to build that on-call AI agent and that was where we felt we got the second biggest ROI. And the third area was how do you make your data ready for AI for agentic analytics? there's a lot of demand for self-serve. We can't build dashboards fast enough for business to get insights from. And probably that is changing and disrupting as we go. We wanted to spend some time to make sure the data is AI ready where people can ask questions themselves and self-serve in a trusted and reliable way. So these three pockets were the areas where we felt we want to really zero in, go deep and use AI to help us accelerate. Ido Bronstein: Yeah. Amazing. So if we will jump to the end and in minute we will take the whole journey. What is the state today? How much you succeed to automate? And what is the return that you see right now in the field? Prakash: Okay, you're gonna put me in trouble here again, but ⁓ it's a journey. Ido Bronstein: No, Prakash: Yeah, it's a journey we've been in. It's been close to six plus months that we've started to kind of invest our time, capacity and synergies here. ⁓ We did not initially block any dedicated time to do AI driven development. This was a coalition of the willing. We wanted to ensure that our business goals are not being put on hold. We wanted to prioritize what business goals are. So whatever people are working today, that needs to happen no matter what. And then there were folks who had Ido Bronstein: cost. Prakash: let's say additional time who wanted to kind of spend come together as a group, like a central working group and build those skills that we can start to test, evaluate and see what works. And now we are doing this entire workshop and training modules where we are trying to bring every data engineer in Atlassian in and show them how you can use AI, right? Everybody was doing in their own way. People were trying to find hundreds of different patterns of using AI. And at some point it became very hard to understand how do we consolidate these efforts. So Now that we have a small group of team members who are building these skills, we are running training programs, we are starting to see that there is a good amount of consolidation happening. People are working with each other to improve the skills and starting to use those things in day-to-day life in all the three categories. So the category one was how do we develop things faster? Right? Like you have a requirement completely documented in a JIRA ticket or somewhere. You just point it to your AI agent and say, help me code this and tell me what you have. Show me a plan. execute for me. Now we are seeing data engineers who are using that in a reliable way. So I could say without being too aggressive, if something used to take us, let's say five days of time in the past to develop tests and do some things, we've definitely seen a significant improvement to reduce that five days to maybe less than three days now, right? Definitely there has been good productivity improvements. Now I try tend to look at it for total cost of ownership. So I am a bit kind of, you ⁓ careful in saying that yes, we are seeing 30, 40 % improvement gains. When we ⁓ use AI to build a code, are we building a scalable code? AI breaks often when the AI generate a code? Does it break often in production? Are we accelerating development, but the operational cost has improved, increased? So I'm trying to look at it from a total cost of ownership standpoint. through that lens, I still see a good improvement since we have. focused on both these pillars. We are trying to use AI to help us intercept the problems faster. We are using AI to now develop our ETL code faster. So we are seeing compounding gains when you have both of them work in parallel. ⁓ So yeah, I can clearly say that I'm definitely seeing 10 plus percentage of productivity gains even at an early stage right now. It's only gonna get better from here as we improve our maturity. Ido Bronstein: Amazing. So let me see that I understood your journey correctly. So in the beginning, you found three areas that you want to focus. Taking small JIRA tickets that you are getting from the business, trying to automate them from like smaller pit of tasks that you think that AI can solve. Second thing is on call, you have an incident, you want to investigate something in your data, you want that AI will do that. AI ready data for the business, being able to query the business in a free language. So anyone in your organization to access the data with AI. In the beginning, try that each one of your team will try to just use AI in order to do those three tasks. You get a lot of different solutions, a lot of different skills, each one got different result. no unification, then you come back, did workshops that create like unified skills, solve challenges that AI has in each one of these different pillars. And today you already seeing, you already seen a boost in your productivity. It's hard to say because the productivity is playing across all the lifestyle and not only on specific job, but you feel that you are in the right direction. Prakash: Yeah. We definitely feel we're in the right direction. I I definitely missed out a lot of details in between. There were small, small things like, hey, how can we improve our data quality of our tables by just adding more data quality checks using AI? We can just point AI to all the metadata we have and say, hey, can you generate some boilerplate default data quality checks that we can catch some of the issues happening? So we've kind of taken a lot of different steps at it through entire SDLC, and we found a lot of good areas where we've sped up our, you know, development process. ⁓ In terms of self-serve, I think the biggest challenge that I see right now is ⁓ data is extremely hard to query through LLMs. People ask questions in many different ways, different ways of English questions people ask. And even whether you use Genie, you use Cortex, or you use, let's say, Cloud Code, whatever pointing to your data, I've seen that. the translation of natural language to the actual SQL, actual semantic layer between your data isn't a hundred percent accurate. Even the Birdbench doesn't say a hundred percent accuracy for AI. It's around 84%. Humans are the most accurate right now at 91%, right? But Birdbench says every other model like Databricks, Gemini, there are 81, 82%. So there is still a gap of how much you can self-serve, how reliably you can self-serve, but there is an investment that... Ido Bronstein: Yeah. Prakash: we as a company at Atlassian are making to improve our context layer, build good metadata, good lineage, improve our catalog maturity. So we are making some significant investments to head in the direction where we build a rich context layer and use that context layer not only to self-serve data insights, but also to build vertical and horizontal AI agents that can enable workflows, right? So we are in that journey right now. Ido Bronstein: Amazing. Yeah. And this is tied back to what we talked about in the beginning. It's always comes back to the foundations without good metadata, lineage, curation data set. You cannot really trust ⁓ they are in the end. And to your question in the beginning, this will always be the work of data engineering. Check that those things in place. Prakash: Yeah, and that's why saying into earlier question, will AI repeat data engineers? I don't think so. Yes, AI will definitely improve the value proposition of a data engineers. Data engineers will not be writing ETL pipelines anymore. Maybe they will write small parts of it or they would probably become more architects and push the value proposition of the role more towards enabling value to the business, value to the company, rather than just kind of being Ido Bronstein: Yeah. Prakash: deep into coding, right? I think that's how the role will evolve, but data engineers will still be very, very crucial to build the core foundations and fundamentals that will unlock the power of AI, yeah. Ido Bronstein: So I'm sure that our audience are data engineers that really want to understand what it means to use AI in data engineering ⁓ works. So if you can share one or two technical challenges that you solved during this journey, I think that it will be very interesting to learn from it. Prakash: Yeah, I think before I go to technical challenge, there's a human challenge here as well. The rate of improvements and developments in AI is so fast that I don't think any one of us can keep a track of what's happening. It's extremely difficult to keep yourself abreast with that. The challenge of driving a huge transformation like AI in an organization where things move really fast is how do you have the right knowledge? delivery mechanism to the people. How can we upskill the workforce? How can we make everyone aware of how can they use AI? Not just by telling them, go figure it out by yourself, learn it by yourself, taking some courses, but providing guided pathways of learning, showing folks examples of how you can step-by-step use different AI skills, AI tools to do day-to-day job, and then see how productive AI can make us, right? I think that step-by-step enablement is extremely important. And then we've seen that challenge in the company. We've brought in a bunch of tools to use. Atlassian has our own AI agent called Robo. It's also a product that we give it to our customers to use on top of Atlassian Suite. We use Robo day in and day out to ask questions, help us to write pages, draw diagrams, write code for us now, all that stuff. So it's a powerful tool. We can connect to any LLM engine underneath the hood. But if we don't tell people how to use it for doing a particular task or job, it becomes dumpster fire where folks do it in different ways. technical challenges, I do like you asked, hallucination is real. It is extremely difficult to get deterministic answers no matter how well you applied guardrails around, given the right instructions, you have set up the skills to be atomic enough to do what it is. When you... Ido Bronstein: Yeah. Prakash: orchestrate and load these skills up into your agent and ask it to do. It will hallucinate three to four times out of every 10 times you ask it to do, but kind of being aware of when did it do the right thing, when did it hallucinate. That's where data engineers or any engineers in general have to become extremely smart to know how to catch these as it's happening and not just take the work that AI has delivered to the face value of that. There has to be some amount of validations that we have to do on top of that. Unless until it gets better, it gets more accurate, it gets more deterministic. I think we will have that problem. ⁓ The other last thing that I want to talk about was organizations who don't encourage people to try new things will eventually slow down in this journey of AI adoption. Right. And one of the reasons why early on we were happy to let people try AI as much as they want, find new patterns, find new ways of solving the problems was we wanted to get people excited about the creativity that they can now show using AI, right? It's really important that you give people the freedom to use whatever LLM tool, whatever technology available to them to find different pathways. And when you have different pathways, then you can find what works, what doesn't work, and you can consolidate and synergize your team to say, let's now find common parts together, right? I think encouraging people to go try different things, I think is extremely beneficial. And organizations who don't do that probably will slow down and probably will see that. lagging effect of AI. Ido Bronstein: So, yeah, so AI needs to be adapt from bottom up. You cannot like tell people like use AI, use AI. No, they need to really use that, understand where it really works. And yeah, and then you need to spread what works in the organization. And I think this is amazing. And actually one of the reasons we decided to open this podcast, the Data Splash, is because we understand that there's so much that is going on and... Data practitioners really need to learn a lot and they don't have like very condensed way to understand how to use that and how to leverage that. trying to leverage this platform for that. So next time your team wants to something new, I hope that they will come to the data splash. Last question. What do you think is the next step? Where is it forward? ⁓ Where do you see Atlassian in year, year and a half from now? Prakash: Yeah, that's a really interesting question. I wish I had a crystal ball to know where we are going. I think anyone's guess would be better than mine right now. From an industry standpoint, I feel 2026, 2027, we will start to see a lot of improvement in the way LLMs work with data, data technologies, thanks to companies like you, you folks who are trying to pioneer this, building great products to make data engineering. data analysis, autonomous. think that's amazing. The way I see it going forward from industry standpoint is many of the data roles that we've had in the past, right? mean, if I decide to take a step back, there is a data engineer, there is a data analyst, there is an analytics engineer, there is a business analyst, there's data scientists, and there's a bunch of other people in the same realm. I definitely will start to, we should start to see these roles kind of converge or kind of unify under one title or two titles, if you will, right? Like a data practitioner or data expert, whatever, I don't know, whatever you call it, right? Because the bar to learning other things that these other also doing is extremely easier now. You can ask a Databricks Genie or Cortex to write a linear regression for you on the data. It can give you all the statistical analysis and all that stuff. So the bar is shortening down. I think the roles will kind of converge at some point in time and... Ido Bronstein: Yeah. Yeah. Exactly. Prakash: It is just that the momentum is building towards that right now. Every company that I see right now are building tools to kind of converge these skill sets. It's making it easier for you to do those things. How reliable? That's where the real question is. But eventually I think we will get to a point where these are very reliable, very deterministic. From an Atlassian standpoint, ⁓ I think, yeah, yeah. Ido Bronstein: Yeah. I just add that I think that I see this a lot in the industry, like this understanding that all the data practices is going to merge. The ways that I see that ⁓ is going to merge to something that we will call it like knowledge engineer, because it's going to be how you craft the knowledge of the organization. And the data is something that's very technical, but what you want to achieve is knowledge engineer and knowledge, and knowledge help organization. So it's going to be cool. I saw that Airbnb just released in the last summit of like the open data forum. They call it like the full stack data engineering. So I think that right now everybody call it in a different name, but like it's going to happen. Like those different jobs is going to consolidate because things, part of the things becoming easier, but also part of the things becoming much more important to craft this knowledge and help you to really enable the organization with it. Prakash: Yeah, and along with that, I've heard some other titles that I've been hearing through Gartner Summit and other places where I was there a few months ago. Context Engineer, Agent, sorry, Agent Evaluators or Agent Orchestrators, where people have to be really good at knowing how to create custom agents to do custom tasks and activities using the skills of all the underneath below roles, right? Ido Bronstein: context. Yeah. Yeah. Prakash: the value proposition is gonna get pushed more upstream towards value creation rather than just doing the work. I think the job roles will itself not go away, but we will definitely, this will encourage all the existing data people to start to think about higher value creation, work with business teams to understand what they can really solve and then use the tools at their disposal to kind of become extremely 10x productive from what they're doing. I think that's where we're heading towards and this will accelerate. I believe for Atlassian, don't know how this will look like, but I think we are inching and gearing towards the same direction where we want to first make existing data engineers extremely productive, give them all the tools and arsenal to kind of meet the needs of the company today and tomorrow, right? And once we are able to do that, we wanna kind of skill the people that we have today to make sure they can understand business concepts better, they can solve business challenges in a more effective way without worrying about the data stack, right? So we will... coach folks to make sure they are more value creators than data builders. I think that's where I would see Atlassian data folks going towards from here. Ido Bronstein: I Yeah, and I hope this future will come soon. Yes. Yes. Prakash: Yeah, can't wait. And I hope that none of the things I said is any controversial and gets me to travel, but it was really fun to kind of discuss and nerd out on some of these concepts. And yeah, I'm really excited to see, I know how your company and your journey is as well building towards this future that we're going towards. So really excited about it. Yeah. Ido Bronstein: No. Yes. first all, Parkesh, thank you much for ⁓ jumping the call with me. ⁓ I know I learned a lot. ⁓ Just a of quick takeaways that I taking from this podcast. The first is ⁓ you need start from a good foundations ⁓ and without good ⁓ without good metadata, ⁓ you cannot scale to ⁓ AI. Second thing, You have to experiment with AI. You need to play with your hand. You need to understand what works, what don't work. Organizations that won't do that today will find themselves irrelevant in two years from now. And lastly, AI is not magic. It's not just plug and play. Especially in the data, there is a lot of know-hows. You need to understand what works in your environment, in your ecosystem, and it's going to take time. So you better to start early. And it's amazing to see how Atlassian took the leap very early and already very progressed in this journey. So very happy to learn from you and very happy that our audience can learn from you. So thank you so much for your time. It was a pleasure. Prakash: Thank you. Thank you for having me.