Latest / Agentic Conversations (formally mlops.community) / Meta GenAI Infra Blog Review // Special MLOps Podcast

Transcript

Meta GenAI Infra Blog Review // Special MLOps Podcast · Agentic Conversations (formally mlops.community)

• 0:04
  At Meta, we handle hundreds of trillions of AI model executions
• 0:09
  per day, delivering these services at.
• 0:12
  A large scale requires. A highly advanced and flexible
• 0:16
  infrastructure custom designing much of our own hardware,
• 0:21
  software and network. Fabrics.
• 0:24
  Allows us to optimise the end to end experience.
• 0:28
  For our AI researchers. While ensuring our data centres.
• 0:33
  Operate efficiently. That was a quote from 1:00.
• 0:37
  Of the three blogs. That I want.
• 0:38
  To talk about today. And we get into so many
• 0:44
  different cool pieces of the meta, generative, AI and ML
• 0:50
  infrastructure that they have built.
• 0:52
  Out over the past. Year, year and a half.
• 1:03
  I want to talk about my observations first, then we will
• 1:08
  go into design pillars from metas.
• 1:10
  Blogs. And the learnings from the large
• 1:14
  training jobs that they're doing, AKA creating Lama Lama
• 1:20
  two, Lama 3, then we can get into their infrastructure stack.
• 1:25
  And last but not least, the maintenance.
• 1:28
  Of how they. Keep it up, been running
• 1:31
  continuously and reliably. The three different blog posts
• 1:36
  that I will be. Referencing.
• 1:37
  For this podcast or? How Mediterraneans large
• 1:42
  language. Models at scale.
• 1:43
  Maintaining. Large scale AI capacity.
• 1:47
  At meta and building metas Gen AI infrastructure.
• 1:52
  This is a. Synthesis and my.
• 1:54
  Key takeaways of all three of those blogs.
• 1:59
  I'll leave the link in the description to each one of these
• 2:02
  blogs in case you want to just go and read.
• 2:04
  It yourself. Which I highly.
• 2:05
  Encourage you to do. Let's get into it first.
• 2:09
  With my observations, one thing is clear.
• 2:12
  After. Scouring the meta blog Meta
• 2:16
  Engineering. Blog, I should say.
• 2:18
  For. The past 2-3 weeks.
• 2:21
  They are constantly thinking about future compatibility.
• 2:27
  They know that whatever size they are now is relatively small
• 2:33
  compared. To what they will.
• 2:35
  Be in the future, they're continuously planning for things
• 2:41
  to just balloon. That whether?
• 2:42
  That is their compute. Their.
• 2:44
  Storage or the number? Of developers.
• 2:46
  Will only grow in size and it seems.
• 2:49
  That they are always. Thinking about scale and how
• 2:53
  things will scale in the future now.
• 2:57
  Of course. Always thinking about scale for
• 3:00
  your company is. A great.
• 3:03
  Thing if you are. Meta but it.
• 3:06
  Reminds me. Of that classic blog post, that
• 3:11
  was. Referenced frequently in the MO
• 3:14
  OBS community, which is you are not Google.
• 3:18
  As. We are walking through.
• 3:20
  All of the cool. Stuff that Meta is doing, just
• 3:24
  try and pinch yourself every once in a while.
• 3:26
  And remember, you are not Google, so there's gonna be some
• 3:30
  actionable, tangible takeaways. And there's going to be some.
• 3:33
  Stuff that's just like, wow, that's cool.
• 3:35
  That they're doing it. Try to.
• 3:37
  Discern between the two and only you know what size and what
• 3:41
  scale. You are at.
• 3:43
  Next observation was. That the price.
• 3:47
  Of downtime for meta is expensive.
• 3:52
  They are constantly talking. About how they try very very
• 3:57
  very hard to keep all. The GPU's online and maximise
• 4:02
  networking speed. And why is that?
• 4:05
  Well, each. Millisecond that is not
• 4:10
  optimised or each minute of downtime.
• 4:13
  Costs a whole boatload of cash. To the company and the
• 4:18
  engineers. Are very aware of this So what
• 4:21
  do they do they're. Thinking about optimising Every
• 4:26
  sliver of optimization translates into dollar signs.
• 4:31
  Think about. If you could.
• 4:34
  Optimise something .0001% and that translated into millions of
• 4:40
  dollars. That's.
• 4:42
  What they're doing with at meta and it is very obvious when you.
• 4:45
  Read these blog posts. They know that, and even with.
• 4:49
  That. When in NVIDIA comes out with a
• 4:51
  new GPU, they as in Meta, not Invidia.
• 4:56
  They have to invest in tailoring it to their environment.
• 5:01
  To truly. Get all the benefits.
• 5:02
  There is a section that I'll go to later on about.
• 5:06
  How they're AB? Testing the different
• 5:10
  networking. Speeds.
• 5:13
  To find. Out which gives better results,
• 5:16
  They look at the whole stack. From top to bottom, as far deep
• 5:23
  as you can go and. They try to shave off little
• 5:27
  parts. On each.
• 5:28
  So they're optimising Pie Torch, they're optimising the
• 5:31
  scheduling, they're optimising the hardware.
• 5:33
  That means like networking, that means that it's the maintenance.
• 5:38
  Of the hardware they're. Optimising storage the data
• 5:42
  centre deployments where they. Deploy the GPUs.
• 5:45
  What the cooling? And the power capabilities are.
• 5:48
  All of these things they're keeping in mind and after
• 5:52
  reading. These blog.
• 5:54
  Posts. It may have been selection bias,
• 5:56
  so it might. Just be that these.
• 5:57
  Three blog posts mention it quite a bit.
• 6:00
  Or it may. Be because it is.
• 6:01
  Very front of mind. It seems like time and time
• 6:04
  again networking is the most important.
• 6:08
  Piece of the puzzle? They are constantly referring to
• 6:11
  how important networking. Is that's.
• 6:15
  The main character here and so I look at this and I think.
• 6:20
  At what cost are? They creating this incredible
• 6:28
  technology. The irony of rushing towards
• 6:32
  something like trying. To create AGI.
• 6:36
  And by the way. They say they are.
• 6:38
  Fully committed to open source, and I believe that they've done
• 6:42
  a lot of. Cool stuff.
• 6:43
  With their technology, like Grand Teton, they open.
• 6:47
  Source that obviously pie. Torch is fully open source, all
• 6:50
  the Llama models are open source and their long term ambition is
• 6:55
  to build open source. AGI, but that makes.
• 7:00
  Me roll my eyes. Let's.
• 7:02
  Put aside the fact. OK, what does?
• 7:04
  AGI even mean? Let's just disregard.
• 7:07
  That whole argument and let's focus on Meta creating AGI Meta
• 7:13
  the company. That is related.
• 7:15
  To selling user data. To the highest bidder.
• 7:18
  The company. That is related to.
• 7:20
  Helping genocide in Burma. The one who covered up internal
• 7:25
  studies on how Instagram causes mental health issues for teens.
• 7:30
  The one who rebranded to Meta, they're the ones that are
• 7:35
  dedicated to creating AGI. Lord help us, but if that's what
• 7:42
  they got to do, that's what they got to do.
• 7:43
  It's easy to judge, so I'll keep it at that.
• 7:46
  Let's go back to this At what? Cost question because as I'm
• 7:50
  reading this and I'm hearing about.
• 7:52
  These. Millions and millions that
• 7:54
  they're spending billions on their AI efforts and all this
• 7:59
  energy. It's just.
• 8:02
  Ironic because they may be using completely sustainable energy.
• 8:08
  Who knows all the energy that they get for their data centres.
• 8:11
  It might be coming from solar power.
• 8:13
  They don't really talk about it, or at least I didn't see a blog.
• 8:15
  About it. But it reminds me of a
• 8:18
  conversation. That I had with.
• 8:20
  Sasha Luccioni, who's now a researcher at Hugging.
• 8:24
  Face but. Back when I talked to her in
• 8:26
  2021, she. Was a researcher at.
• 8:29
  A university and she mentioned. There are these great.
• 8:35
  Feats of computing or these incredible things that we've
• 8:40
  been able to make. Computers do.
• 8:42
  Like in that specific conversation she referenced a
• 8:47
  robot being. Able to put together a Rubik's
• 8:50
  Cube. But we are not having any type
• 8:54
  of conversation at how. Horrible.
• 8:58
  A waste. Of energy, that is to train a
• 9:01
  model we spent so much energy being able to.
• 9:06
  Create a robot. That can now put.
• 9:08
  Together a Rubik's Cube. Great job, but at what cost?
• 9:14
  And so those questions are not being raised.
• 9:16
  I have to raise them here in the observation section.
• 9:21
  Let's get into how Mediterraneans LLM's.
• 9:23
  At scale, so the. First thing that we have to note
• 9:27
  are the differences between looms and traditional machine
• 9:32
  learning at Meta, which was primarily recommender systems.
• 9:36
  Think about recommender systems and how you interact with Meta
• 9:39
  and they are pretty. Revolent across the whole.
• 9:45
  Platform, whether that is. Facebook.
• 9:47
  Ads or Instagram ads? Your home feed search
• 9:52
  marketplace search marketplace in general people you know.
• 9:56
  It's all recommender systems and now limbs are almost like a 180.
• 10:04
  Because they require way. More compute and it's fewer.
• 10:10
  Jobs. So Rexes.
• 10:13
  Needs to be like very fast and there's.
• 10:16
  A lot of small. Jobs that are constantly taking
• 10:19
  place like Boom show you an ad. Boom show you a person you might
• 10:23
  know. Boom, a new video on your feed.
• 10:26
  But now these LM's. When you're training something
• 10:30
  like Llama, you need way more compute and.
• 10:35
  So that creates. Lots of GPU problems.
• 10:40
  To solve. That the meta team has some
• 10:44
  performance design pillars when they built out their training
• 10:49
  infrastructure, these four design.
• 10:52
  Pillars are hardware reliability.
• 10:54
  Fast recovery on failure, efficient preservation of the
• 10:59
  training state and optimal connectivity between GPU's I
• 11:06
  think. It is pretty self.
• 11:07
  Explanatory what each one of. These are, but let's.
• 11:10
  Break them down a little bit more.
• 11:12
  Hardware reliability is. And I quote from the blog
• 11:15
  ensuring that. Our hardware is reliable.
• 11:18
  Is important. We need to minimise the chances
• 11:23
  of a hardware failure interrupting a training job.
• 11:26
  This involves rigorous. Testing and quality control
• 11:29
  measures and automation to quickly detect and remediate.
• 11:35
  Issues. Then you have fast recovery on.
• 11:38
  Failure despite our best efforts.
• 11:40
  Hardware failures can and do occur.
• 11:43
  When they do, we. Need to be able.
• 11:45
  To recover quickly. This involves reducing
• 11:48
  rescheduling overhead and fast training reinitialization.
• 11:56
  Next up is efficient preservation of the training
• 12:00
  state. In the event of a failure.
• 12:02
  We need to be able. To pick up where we left off.
• 12:04
  This means we need to regularly checkpoint our training state
• 12:08
  and efficiently store and retrieve training data.
• 12:11
  Last but not least, optimal connectivity between GPUs.
• 12:16
  Large scale model training involves transferring vast
• 12:19
  amounts of data between GPUs in a synchronised fashion.
• 12:23
  A slow data exchange between a. Subset of GPUs.
• 12:26
  Can compound and slow down the whole job.
• 12:29
  Solving this problem requires a robust and high speed network.
• 12:34
  Infrastructure as well as. Efficient data transfer
• 12:38
  protocols and algorithms so. Basically, if we walk through
• 12:42
  this step by step. They're saying.
• 12:45
  We're going to do our. Best to make.
• 12:47
  Sure that the hardware doesn't fuck up, but if it does fuck up.
• 12:50
  We're going to make sure that we can.
• 12:53
  Reboot it. Really.
• 12:54
  Really. Fast and.
• 12:56
  Hopefully we catch it as quickly as possible because.
• 12:59
  Again, each second of downtime is costing millions.
• 13:04
  And thirdly, we want to make sure that the training jobs can
• 13:09
  get back up and running as quickly as.
• 13:12
  Possible. If something does go down, so
• 13:15
  we're going to preserve the training state and last but not
• 13:18
  least, we want to optimise. The throughput of the GPUs
• 13:22
  because if you have the. Slightest amount of slowdown
• 13:27
  across the 24,000 or 54,000 or 128,000.
• 13:33
  GPU that they have. That's going to compound and be
• 13:37
  painful for everyone that's using the platform.
• 13:40
  Along this note of reliability. They call out clearly.
• 13:45
  That the number of failures increases.
• 13:48
  With the cluster size. It's an inevitable and some of.
• 13:52
  The top hardware failures. That happen are because.
• 13:57
  Of. GPU's falling off.
• 13:58
  The bus a GPU driver is stuck. You have.
• 14:03
  DRAM, UCE, SRAM, UCE, network cable failure.
• 14:09
  I mean there's a few things. Let's let's break these down 1
• 14:14
  by 1. GPU's falling off.
• 14:16
  In this case GPU are not. Detected by the host.
• 14:19
  On. PCI.
• 14:22
  There are several reasons. For this failure.
• 14:24
  But this failure mode is seen more in the early life and
• 14:29
  settles. As the server ages.
• 14:32
  So when you get those brand new H1 hundreds out, boom, you
• 14:36
  start. To see GP.
• 14:37
  'S falling off a little. Bit because it's like.
• 14:38
  What the hell is going on here? And after some time and the
• 14:43
  server settles, you don't see that as much when it comes to
• 14:46
  DRAM and SRAM. You see, uncorrectable errors
• 14:50
  are common in memories, and we monitor and identify repeat
• 14:56
  offenders, track against thresholds, and initiate RMA's
• 15:00
  when error rates exceed vendor. Thresholds.
• 15:04
  Last but not least. The good old.
• 15:05
  Hardware, network, cable. This is very interesting to me
• 15:10
  because. It's like I just.
• 15:12
  Envisioned someone in the data centre with some wire cutters
• 15:18
  cutting something and pissing. A few researchers off.
• 15:21
  But getting back on track that doesn't they do not say that.
• 15:24
  To be clear, they do not say that anywhere in the blogs.
• 15:27
  But that's just my vision when I read hardware, network cable
• 15:32
  failures in the general category of unreachable servers.
• 15:36
  These failures are also. Seen most often in the early
• 15:40
  life of a server. So when it comes to training
• 15:43
  LM's for meta performance and reliability is.
• 15:47
  Crucial they are. Focused on building end to end
• 15:52
  AI systems with a major emphasis on researcher and developer
• 15:57
  experience and productivity and the ways.
• 16:00
  That they do that are. Custom designing their own
• 16:04
  hardware, software and network fabrics.
• 16:07
  Speaking of networking, I told. You.
• 16:09
  It was the star. Player in these different blogs,
• 16:12
  they are testing two different systems and it is fascinating to
• 16:18
  me because both of these systems.
• 16:20
  Are like way bigger clusters. Than any.
• 16:24
  Of us will probably ever have access.
• 16:26
  To in our lives. But they just.
• 16:28
  Talk about it nonchalantly. These two.
• 16:32
  Different clusters are of 24,000.
• 16:36
  GPU S so they. Have in total 48,000 GPUs and
• 16:43
  there's one 24,000 GPU. Cluster that has one.
• 16:46
  Style of networking and another 24,000 that has.
• 16:51
  The other style, so the 1st. Style that I want to.
• 16:54
  Talk about is. On cluster remote direct memory.
• 16:59
  Access. Over converged Ethernet network.
• 17:04
  Fabric. Solutions ROCE and it's based on
• 17:08
  the Arista 7800 which is a data centre switch router.
• 17:14
  It also has the wedge. 400 and Mini.
• 17:18
  Pack two OCP rack switches. So for all you hardware geeks
• 17:23
  out there, I guess that's really cool.
• 17:25
  I don't know enough about it to say anything, but it's just
• 17:28
  breaking down all of. The physical pieces.
• 17:33
  In the rack, some of the highlights.
• 17:34
  Of the arista for. Those who want to know it has up
• 17:39
  to 96 billion packets per second and.
• 17:43
  Fully scheduled lossless architecture.
• 17:46
  Cell spraying fabric for 100% efficiency, fully scheduled
• 17:51
  credit control, high hardware availability, dual input grid,
• 17:56
  redundant power supplies, cloud grade routing.
• 18:01
  So the list kind of goes on and on and it's nice that they're
• 18:04
  giving them a little shout out that they're using that inside
• 18:07
  of the data centre. On the other hand, the other
• 18:09
  24,000 GPU cluster is using Nvidia's Quantum to InfiniBand
• 18:17
  fabric, and I think that's what most of us are probably more
• 18:21
  familiar. With because.
• 18:23
  That's the more popular one. Or at least.
• 18:26
  That's the one that I hear about more often in the blog post
• 18:30
  where they're going deep on their.
• 18:32
  Hardware. Architecture, and specifically
• 18:34
  their networking architecture and their.
• 18:37
  Choices they. Reference some of the trade-offs
• 18:40
  and some of the design decisions that they had to make when they
• 18:46
  were building out the networking and.
• 18:47
  Why they chose to create 2? Different 24,000.
• 18:53
  GPU clusters to A, B test what was better.
• 18:57
  So I'm going to read that right now.
• 18:58
  I'll read you. What they say.
• 19:00
  Meta had built. Roc, E Clusters.
• 19:04
  And I'm just going to pronounce. ROCE.
• 19:06
  If later somebody tells me that it's pronounced Rosie.
• 19:10
  I would love that. But for now, we're going to
• 19:11
  pronounce RCEP meta had. Built Roc E clusters for the
• 19:16
  past four years, but the largest of those clusters.
• 19:19
  Only supported 4000 GPUs we needed we Meta needed
• 19:24
  significantly larger Roc clusters.
• 19:28
  On the other hand, Meta had built research clusters with
• 19:31
  InfiniBand as large as 16,000 GPUs.
• 19:34
  However, those clusters. Were not tightly integrated into
• 19:37
  Metis production environment nor.
• 19:40
  Were they built for? The latest generation of GPU's
• 19:44
  networking. This made for a difficult
• 19:46
  decision. Of what fabric to build with.
• 19:50
  So we decided to build with both.
• 19:54
  Lucky them, they can just do that.
• 19:55
  That's incredible. Two. 24,000 GPU.
• 20:00
  Clusters one with RCEP. And another with InfiniBand.
• 20:04
  Our intent was to build and learn from the operational
• 20:08
  experience. More on testing and prod later.
• 20:11
  These. Learnings will inform the future
• 20:14
  direction of Gen AI fabrics. At Meta we optimise.
• 20:18
  The Roc cluster for. Quick build time and the
• 20:21
  InfiniBand cluster for full by sectional bandwidth.
• 20:25
  We used. Both InfiniBand and.
• 20:27
  Roc clusters. To train Loma.
• 20:29
  Three with the Roc. Cluster used for training the
• 20:33
  largest model. Despite the.
• 20:36
  Underlying network technology differences between these
• 20:40
  clusters, we were able to tune both of them to provide
• 20:44
  equivalent performance. For these.
• 20:46
  Large Gen AI workloads. This is really cool to see
• 20:51
  because. It's almost.
• 20:51
  Like they're curiosity. To know if they could be doing
• 20:55
  it better made them. Take the hard route.
• 20:58
  And say you know. What?
• 21:00
  It's probably easier. To do the InfiniBand, but let's
• 21:03
  see if there's something. Better out there Let's.
• 21:05
  Roll our own just in case we can shave off a few milliseconds for
• 21:11
  their Gen AI infrastructure. All of the.
• 21:14
  Computer is built with. The Open.
• 21:17
  Source project they created. Called Grand Teton, they've been
• 21:21
  openly designing. The GPU hardware.
• 21:24
  Platform beginning. With the big surf.
• 21:26
  Platform in 2015 and anyone that does not know the big surf
• 21:32
  platform, there's a whole blog post on Facebook to open.
• 21:36
  Source AI hardware. Design and this.
• 21:40
  Is the big serve. Platform.
• 21:42
  This was a 2015 initiative. So are they?
• 21:48
  Spending as much time, energy and resources on R and.
• 21:52
  D to keep this up to date this. Big surf platform or is this
• 21:58
  just a snapshot in time and it hasn't?
• 22:01
  Really evolved. Since 2015, they don't go into
• 22:04
  the detail, but that is one of the questions that I had.
• 22:07
  Is like. How much are they trying to
• 22:10
  optimise this piece? I imagine if they have engineers
• 22:14
  that are playing around with these two different types of
• 22:17
  networking, they've probably got engineers that are figuring.
• 22:21
  Out if the big. Serve Platform or Grand Teton
• 22:25
  can be optimised, but if not and I just gave them all.
• 22:29
  A great idea. To shave off some milliseconds.
• 22:33
  OK, that's their compute. We've talked about the
• 22:35
  networking. We've also talked about.
• 22:36
  Design pillars. Now let's go into.
• 22:39
  The. Storage for the Gen AI platform.
• 22:42
  One thing that is very clear, these folks are not using Duck
• 22:46
  DB for anything. The size of the storage.
• 22:51
  That they need. Holy smokes, it reminds me.
• 22:54
  Of a talk that I just. Saw from Chang from Lance DB and
• 23:00
  he was talking about how there has been this rise.
• 23:04
  Of. Data engineers saying big data
• 23:07
  is dead unless. You work with video or photos
• 23:12
  because if you work with video or photos, big data is alive.
• 23:17
  And well, so. All you out.
• 23:20
  There that are doing multimodal stuff you know big data is not
• 23:24
  dead yet and this is the Facebook team or the media team
• 23:28
  they. Know that so.
• 23:30
  Training these large models can be an absolute mess I want to
• 23:36
  reference. Two other podcasts that we've
• 23:39
  had where. We get into details about the
• 23:43
  way things can. Go wrong.
• 23:45
  Specifically, we. Had one all around checkpointing
• 23:50
  and how to checkpoint correctly or?
• 23:53
  What a headache. Checkpointing can be when you're
• 23:56
  training LLM's, and that was with Nebulous AI.
• 24:00
  Simon came on and talked to me all about it and his learnings
• 24:06
  as he was trying to checkpoint. Terabytes worth of data.
• 24:10
  I just saw Todd. Underwood, who is.
• 24:13
  Working on the researcher platform at Open AI at the AI
• 24:18
  Quality conference and one of the things.
• 24:20
  He said is. How?
• 24:22
  Difficult it is. To know when to checkpoint and
• 24:26
  how to checkpoint because it's not like you get some guide and
• 24:29
  it says checkpoint. Every three steps.
• 24:32
  And you'll be good because maybe sometimes you can go 500 steps
• 24:36
  and you're fine, and then other times you can go 2 steps and
• 24:40
  you're not fine. So it's varied.
• 24:42
  In the amount of steps that you can do and you always want to
• 24:47
  make sure that you don't checkpoint too often.
• 24:52
  But you do it. Enough so that if you're
• 24:55
  training for six hours and suddenly something crashes, you
• 25:01
  don't. Lose that six hours.
• 25:03
  Of training. So it's a bit of a dark.
• 25:06
  Arts I would say in the checkpointing and then the.
• 25:09
  Other. Episode that I'll reference to
• 25:11
  talk about how much of a mess training large language.
• 25:15
  Models can be is. The one.
• 25:17
  That we had with. Davis about building DB.
• 25:21
  RX the large. English model from the Mosaic
• 25:24
  team. Mosaic AKA Databricks team and.
• 25:28
  He talks all about how you try to.
• 25:30
  Debug code and maybe it's the. GPUs that are the.
• 25:34
  Bottleneck maybe? It's the data that's.
• 25:36
  Actually not clean. Or you're referencing a table
• 25:39
  that didn't get added into the data.
• 25:41
  Or maybe it's that there's a new version?
• 25:44
  Of Pytorch. But you haven't.
• 25:45
  Patched it yet? Or.
• 25:46
  You did patch it and then it's causing problems downstream.
• 25:49
  There's just so many things that can go wrong and.
• 25:52
  He. Explains them all eloquently.
• 25:56
  Now back to Meta and how they're doing the AI checkpoints and
• 26:02
  specifically how they're storing their checkpoints.
• 26:06
  What they decided to create was. A homegrown Linux file system
• 26:12
  in. Userspace.
• 26:14
  It's called FUSE. FUSE.
• 26:18
  And So what? They did is.
• 26:22
  There's an. API backed by a.
• 26:24
  Version of Metis Tectonic distributed storage solution and
• 26:30
  it's optimised. For.
• 26:32
  Flash media in this. Part of the blog they talk about
• 26:36
  how they're. Using a tool called Hammer Space
• 26:39
  and they give them a clear. Shout out.
• 26:41
  And one thing that I thought when I read.
• 26:46
  That was whoever was the. Salesperson that negotiated for
• 26:51
  media to give them a shout out in this blog needs a fucking
• 26:55
  trophy. Imagine Meta publicly stating
• 26:59
  that they partnered with you and you are the most reliable and
• 27:02
  fastest and bestest. I hope they're making a lot of
• 27:08
  money from that one so. Basically, it's very clear that.
• 27:13
  Most of what meta does is homegrown because of their.
• 27:17
  Scale but. They created and they.
• 27:21
  Worked. With Hammer Space on this one
• 27:23
  they Co created a networking file system for the developer
• 27:28
  experience what Hammerspace enables engineers.
• 27:32
  To do is. Perform interactive debugging
• 27:36
  for jobs using thousands of GPU. 'S.
• 27:39
  Has code changes are immediately accessible to all nodes within
• 27:44
  the environment, so it's instantaneous and.
• 27:50
  They got a. Huge shout out.
• 27:51
  So congratulations to him in. Space.
• 27:53
  That's awesome. On the performance side of
• 27:56
  things. To keep it.
• 27:59
  Reliable. I mentioned it before.
• 28:01
  They test in prod simulations only.
• 28:04
  Go so far, they just have to test and.
• 28:07
  Prod. They mentioned that they're
• 28:09
  continually evolving. Π torch for higher.
• 28:12
  Scale and. Larger AI workloads they have.
• 28:16
  Scaling issues in what works on a small cluster, quote unquote
• 28:20
  small cluster of 100. GPU's it just.
• 28:24
  Is not. So clear if it will work.
• 28:27
  As you start getting to 1000 GPUs or 10,000, they're
• 28:31
  constantly having to test in prod what that inevitably means.
• 28:36
  Is as we talked about before. There's going to be downtime
• 28:41
  they're planning and they're trying as hard as they can to
• 28:44
  make sure that there's not, but they're inevitably is.
• 28:48
  One way that they try to make sure there is not is.
• 28:52
  They have. Large scale maintenance and.
• 28:55
  So there's a whole blog post. On how they maintain these
• 29:01
  gigantic GPU clusters. So remember back when we said
• 29:07
  the? Majority of Facebook.
• 29:09
  'S money or Instagram's money? WhatsApp.
• 29:13
  I don't know if they make money but the majority of metas money
• 29:16
  is from traditional ML. It's from recommender systems,
• 29:20
  it's from serving you ads really, but they're
• 29:23
  transitioning to supporting these Gen AI.
• 29:27
  Workloads. It's like.
• 29:29
  R&D is happening. To see if.
• 29:31
  Anything can go from here and. As we said.
• 29:34
  They're committed to open. Source AGI what they saw is.
• 29:40
  They need to transform. The whole way they.
• 29:45
  Do things. Because of AI workloads.
• 29:48
  If you compare traditional ML and how that?
• 29:52
  Works versus the AI workloads. AI workloads.
• 29:55
  You're spending so much time training with traditional ML,
• 30:00
  you don't. Get that they?
• 30:02
  Have training jobs that can vary from a few hours, a few minutes.
• 30:08
  A few days. A few months and the training
• 30:12
  jobs can be as small as a single GPU running for a couple of
• 30:16
  minutes, or it can have trillions of parameters and
• 30:20
  often spanned thousands of hosts that need to work together and
• 30:25
  are very sensitive to interruptions.
• 30:28
  In this transition from the traditional ML to the Gen AI
• 30:33
  style of doing things and maintenance and infrastructure
• 30:38
  around it. What they saw was that there.
• 30:43
  Are a few main. Characteristics of GPU.
• 30:46
  Training and these. Characteristics are.
• 30:48
  Demanding so the team has to make.
• 30:52
  Sure to be aware of each one of. These these main characteristics
• 30:57
  are the capacity guarantees some jobs can be.
• 31:03
  Paused. But a lot of meta jobs are time
• 31:06
  critical and recurring or online.
• 31:10
  That means that they can't take large amounts of capacity
• 31:14
  offline on a default basis. Bad hosts.
• 31:19
  Are very, very bad. Many jobs require all hosts to
• 31:24
  be synchronised, especially when they're training these AI.
• 31:28
  Models. The bad hosts are a bit slower.
• 31:31
  And have some non. Fatal hardware.
• 31:34
  Or have networking issues and this is extremely damaging to
• 31:39
  not. Just that.
• 31:41
  Workload. But then like we mentioned
• 31:43
  before, things can domino and pretty soon it's clogging up the
• 31:49
  pipes. There's also low.
• 31:52
  Interruption rate since many. Hosts will work with each other.
• 31:57
  On a shared problem, AI training jobs are sensitive to
• 32:02
  interruptions. And a few other characteristics
• 32:05
  of. GPU training or rollout safety.
• 32:08
  This means the. AI software stack is deep.
• 32:11
  And problems are often hard to pinpoint.
• 32:14
  Remember when I was talking about Davis and the DB RX
• 32:19
  training? This is exactly that.
• 32:22
  We need to be careful when rolling out new components.
• 32:28
  AI training jobs are in general cross host and while.
• 32:32
  Outside of the CUDA. Versions.
• 32:35
  There are rarely hard. Incompatibilities.
• 32:39
  We have learned that cluster consistency is highly important
• 32:43
  to debugging. Maybe that'll.
• 32:45
  Take years off of. Someone's life, so hopefully you
• 32:49
  do not have to. Debug because of that, so to
• 32:51
  give you. An idea of what they're working
• 32:54
  with, especially when they're trying to maintain these
• 32:59
  gigantic GPU. Clusters they have. 30
• 33:03
  maintenance operations going. They have 50 different
• 33:06
  components. That are updated 3.
• 33:09
  Different host verification tasks to ensure optimal
• 33:12
  performance and stability. Thousands.
• 33:15
  Of disruptive AI host tasks every day.
• 33:20
  This is what they they just get. Thrown that on a daily basis,
• 33:24
  the team has to make sure. To keep it.
• 33:28
  Reliable as that shit's going on one.
• 33:31
  Difficulty that they deal with. Is upgrading firmware when
• 33:37
  they're training something like Llama Three?
• 33:39
  And imagine you're using a whole lot of those.
• 33:42
  GPU's except uh ohh. We got up Rd some firmware and
• 33:48
  this is constantly happening. They say.
• 33:50
  It's not just like once in a while that happens, it's
• 33:54
  continuously a thing that they're dealing with and the
• 33:57
  way. That they deal with it is.
• 33:59
  By using maintenance trains. Which is they?
• 34:03
  Shut down a small. Number of GPUs.
• 34:05
  In a cyclical fashion, so it's almost like a train that goes by
• 34:11
  it, shuts down it. Reboots.
• 34:12
  It gets that. Firmware update and then boom,
• 34:15
  it's back online. They have strict SLO.
• 34:18
  'S and it can get very complicated when they're
• 34:22
  upgrading CUDA versions. The lower level software.
• 34:26
  Is always consistent, which makes a tricky problem because
• 34:31
  you can't have. Half of the cluster.
• 34:33
  On one lower level software and then the other half on another.
• 34:37
  Otherwise the training job is going to be a mess.
• 34:40
  As if it wasn't complicated enough already.
• 34:43
  So they've added tooling for rare compatibility breaking
• 34:48
  upgrades, and one way to ensure optimal AI performance was to
• 34:53
  work. With.
• 34:54
  The AI teams to design the optimal size of maintenance
• 34:58
  domain. What does that mean?
• 35:01
  A maintenance domain is the percentage.
• 35:04
  Of capacity they take. Down in one go and they.
• 35:08
  Show a graph here which is. Basically on the.
• 35:13
  Y axis. You've got the maintenance cost
• 35:16
  and on the. X axis you've.
• 35:17
  Got the maintenance domain size and there is.
• 35:22
  A U-shaped graph. On the far.
• 35:26
  Left it says that. There's more interruptions and
• 35:30
  it's very high in cost and it's very low in maintenance domain
• 35:36
  size and as you go more towards the right and the maintenance
• 35:40
  domain. Size gets bigger.
• 35:43
  The maintenance cost goes down in this.
• 35:45
  U Shape but at a certain. Point you hit the rate of
• 35:51
  diminishing returns and then it starts going back up and later
• 35:56
  you have high maintenance domain size and high.
• 35:59
  Cost. So what they did is.
• 36:02
  They have something that they called an OPS planar work
• 36:06
  orchestrator. It can handle get this a million
• 36:11
  operations per day and what it focuses on or what it makes the
• 36:16
  engineers lives easier by doing is it works on overlapping
• 36:22
  scopes of operations and. It will correctly serialise.
• 36:27
  Them. So you have.
• 36:29
  One workload that's here and it'll serialise it and then it
• 36:32
  will make sure that OK when we're doing some kind of a
• 36:36
  update everything. With that.
• 36:40
  Workload is going to get done and it's.
• 36:43
  Serialised here. And then we can upload firmware
• 36:47
  in between the next. Workload if you have.
• 36:51
  To move to a new CUDA version, you may need.
• 36:55
  All of the. Capacity on a new driver
• 36:57
  version. This is really.
• 37:00
  Difficult in an environment with thousands of hosts.
• 37:03
  So it seems like the OPS planner.
• 37:05
  Is the silver bullet. That they've found to
• 37:09
  continuously keep these. GPU's.
• 37:13
  The fresh and going Burr as. The researchers like to.
• 37:18
  Say one thing that the UPS planner also does.
• 37:21
  Is they have this safety switch right there's.
• 37:25
  An auto stop of maintenance trains if maintenance or.
• 37:29
  Failure buffers are exhausted. And so that automatically off
• 37:35
  boards the failing upgrades. So you make sure that you only
• 37:41
  have well tested changes reaching the global system
• 37:46
  instead of one bug getting put into all 24,000.
• 37:52
  GPU's. And there you have it.
• 37:56
  That is the end of. These three blog posts.
• 37:59
  If you have any feedback or you have something that I did not
• 38:04
  include in here that you think is worth me talking about.
• 38:07
  Feel free to leave feedback. So that I can hear it and know
• 38:12
  what to make better for the next time.
• 38:15
  As usual, I'm your host Dimitrios, and this has been
• 38:19
  another edition of the ML OPS Community Podcast.
• 38:23
  I did this one because I got so much.
• 38:25
  Great feedback from the Uber podcast where I.
• 38:29
  Broke down. Their blog.
• 38:31
  Post so I wanted to make another one.
• 38:34
  Huge shout out to. Everyone that gave me feedback
• 38:38
  on that episode much? Appreciated, I always love.
• 38:41
  Hearing it. So if you've got anything to say
• 38:43
  on this episode, let me know.