Latest / What the Hoot! / ActiveMQ Artemis vs. ActiveMQ Classic vs. RabbitMQ: Benchmark Insights | Part 1

Transcript

ActiveMQ Artemis vs. ActiveMQ Classic vs. RabbitMQ: Benchmark Insights | Part 1 · What the Hoot!

• 0:10
  SPEAKER_01: Hello everyone and welcome back to another episode of What's the Hoot here with MeshIQ.
• 0:15
  SPEAKER_01: Thank you again for joining us.
• 0:16
  SPEAKER_01: Today we're digging into the new MeshIQ Research Report comparing three widely used open source messaging brokers: Apache Active MQ Artemis, Apache ActiveMQ Classic, and RabbitMQ.
• 0:27
  SPEAKER_01: And rather than simply asking which broker is faster, this research takes a much more practical approach.
• 0:33
  SPEAKER_01: The testing looks at how these brokers behave under different workload shapes, durability requirements, concurrency levels, messaging patterns, and protocols, including JMS, native clients, and AMQP 1.0.
• 0:46
  SPEAKER_01: That matters because there really isn't a single benchmark number that tells you whether a broker is right for your environment.
• 0:53
  SPEAKER_01: Performance can change dramatically depending on whether you're optimizing throughput, latency, durability, fan out, interoperability, or even operational simplicity.
• 1:01
  SPEAKER_01: So today we're gonna unpack not just what the numbers say, but why the brokers behave differently, what the testing methodology tells us, and how teams should apply these findings when making architecture and production decisions.
• 1:16
  SPEAKER_01: And today, joining me, Dan and JB.
• 1:19
  SPEAKER_01: Uh Dan is our DevOps engineer from SQ, one of our rock stars, and JP Onifra, I'm sure many of you guys know, director uh of the Apache Foundation.
• 1:28
  SPEAKER_01: Um, so really some some really hard-hitting and uh uh wonderful people here with me today.
• 1:33
  SPEAKER_01: So they're gonna help us walk through some of the methodologies, uh the headline findings, some of the surprises in the data that they found, and the practical implications for the teams running these technologies in productions.
• 1:43
  SPEAKER_01: So, Dan, JB, thank you both so much for joining us.
• 1:46
  SPEAKER_01: Thank you for your time.
• 1:47
  SPEAKER_01: Uh, and before we get into the benchmark results themselves, I think the best place to start is with why Mesh IQ decided to revisit the research in the first place.
• 1:56
  SPEAKER_01: So before we start taking, you know, talking about latency and throughput and commits and product protocols, let's establish the reason for doing this work.
• 2:04
  SPEAKER_01: Um, and so before we get into the data, what what prompted uh Dan?
• 2:08
  SPEAKER_01: What prompted Mesh IQ to put this report together?
• 2:11
  SPEAKER_01: A what was the gap uh or the question in the market that made this feel like it was worth doing right now?
• 2:17
  SPEAKER_02: Uh yeah, so the original seed for this project, I'd say, germinated as an answer to a rather vague question or situation that we see becoming more and more common in the market today or the field today.
• 2:30
  SPEAKER_02: The problem is more or less summed up as hi, we're tied up with proprietary software X.
• 2:36
  SPEAKER_02: We'd like to explore options and moving away from X, specifically maybe open source options like Y or Z.
• 2:43
  SPEAKER_02: Now, what drives a question like that, right?
• 2:46
  SPEAKER_02: Is it cost?
• 2:47
  SPEAKER_02: Is it return on investment, future proofing?
• 2:50
  SPEAKER_02: There's no shortage of various paths that ultimately lead to the same questions and discussions.
• 2:56
  SPEAKER_02: What else is out there?
• 2:57
  SPEAKER_02: We felt that the time was better than ever to produce a report like this, mostly to mend that knowledge gap, uh, due to a combination of factors.
• 3:07
  SPEAKER_02: We've had increasing inquiries, we see the prevalence and emergence of various open source messaging platforms and technologies, and to go along with that, the various commercial offerings surrounding these open source technologies, or, hey, in some cases, even the buyout of these commercial offerings.
• 3:25
  SPEAKER_02: The field and options today continue to evolve, and we find that it is our due diligence to research and test these technologies, if anything, to offer insight and guidance to any folks who find themselves asking the same question that we see repeated today.
• 3:41
  SPEAKER_01: Love it.
• 3:41
  SPEAKER_01: Uh, and this isn't the first time MeshIQ has done this type of research.
• 3:44
  SPEAKER_01: There was a broader benchmark back in 2022, I believe.
• 3:49
  SPEAKER_01: But the scope of this is noticeably different.
• 3:51
  SPEAKER_01: This this is a follow-up to that 2022 report that covered seven brokers, and this one narrows down to three Artemis Classic uh and Robin MQ, but it goes deeper on each.
• 4:03
  SPEAKER_01: Walk us through that decision.
• 4:04
  SPEAKER_01: What made the team want to go narrower and then deeper this time instead of broader?
• 4:10
  SPEAKER_02: Sure.
• 4:10
  SPEAKER_02: Yeah, this is you know, definitely a follow-up.
• 4:12
  SPEAKER_02: I would say definitely in the nature of the work, definitely in name, but there are uh, you know, as you see, a lot of differences here, right?
• 4:19
  SPEAKER_02: So this was a decision that was made with much deliberation as well.
• 4:23
  SPEAKER_02: Um, I would say I believe a fair way to summarize the the previous 2022 report is that it contains depth, a myriad of different tests and technologies, but ultimately lacked a cohesive direction.
• 4:36
  SPEAKER_02: The 2022 report had so many different tests and parameters, there's no doubt about that.
• 4:41
  SPEAKER_02: Yet at some points I really felt, especially in retrospect, it feels like an ocean that just has the depth of a puddle.
• 4:48
  SPEAKER_02: This report instead narrowed down on a very specific foundational idea that we considered infrangible.
• 4:55
  SPEAKER_02: Every message that is sent by a producer must be consumed, acknowledged by a consumer.
• 5:00
  SPEAKER_02: This is guaranteed end-to-end delivery with zero message loss.
• 5:04
  SPEAKER_02: That was the very idea that kind of drove the testing framework for all the results found within the 2026 report.
• 5:10
  SPEAKER_02: Um, as for the reduced field, we isolated the trio primarily for their open source nature, plus the ever-growing number of enterprises and companies switching to or heavily consider heavily considering switching to these technologies, not to mention all the other larger enterprises that are now offering commercial support for these otherwise open source technologies.
• 5:31
  SPEAKER_02: Right.
• 5:31
  SPEAKER_01: I I appreciate that.
• 5:32
  SPEAKER_01: And I think going deeper also gives you the chance to uncover behaviors that disappear inside broader comparison.
• 5:38
  SPEAKER_01: So, you know, before we we break the findings apart one by one, let's I want to get your your first reaction to the finished data.
• 5:44
  SPEAKER_01: I'd like to open this up to both you and JB.
• 5:46
  SPEAKER_01: So maybe we'll start with JB and then and then Dan you can bring us back.
• 5:49
  SPEAKER_01: But before we dig into the ecosystem level questions, you've had a chance to look at Mesh IQ's findings here.
• 5:55
  SPEAKER_01: What was your initial reaction?
• 5:57
  SPEAKER_01: Anything that confirmed what you'd expect to see, or anything that genuinely surprised you?
• 6:02
  SPEAKER_01: JB, please take us away.
• 6:04
  SPEAKER_00: Yeah, thanks, Mike.
• 6:06
  SPEAKER_00: Um, first of all, uh, I was really impressed when I received a report about the the all the details and all the careful procedure to investigate and to test and to benchmark.
• 6:21
  SPEAKER_00: I mean, it's it's obvious that the mesh IQ team spends a lot of time and effort to really do a great exercise because obviously it's not easy to do benchmark, it's not easy to compare different systems.
• 6:37
  SPEAKER_00: Uh, and so you have to to get really a clear path.
• 6:41
  SPEAKER_00: And the reason why I'm saying that is because also we we see that this report started months ago in terms of investigation, because even the names, you know, we're using Active MQ Classic in the report, but it's not exist anymore now, it's Apache Active MQ directly and Apache RTM.
• 7:01
  SPEAKER_00: So it means that really the the the team spent a lot of time and and it wasn't just not a one-shot.
• 7:08
  SPEAKER_00: And what I like the the most in the report is they are, I mean, I would say confirmation of things we all we we know more or less, but they we we also found some surprises.
• 7:22
  SPEAKER_00: Uh I mean, at least for me.
• 7:24
  SPEAKER_00: Uh for instance, um it was obvious that Artemis is dominating the small message uh durable GMS thing.
• 7:34
  SPEAKER_00: And it was not a surprise to me because it's basically related to the design of Artemis itself.
• 7:41
  SPEAKER_00: Um Artemis is based on append only journal and batch group commits.
• 7:47
  SPEAKER_00: So with small messages, it's where you get the most of you know performances benefits.
• 7:55
  SPEAKER_00: Uh on the other hand, on Active MQ, um we clearly see that with a batch size of one, you are basically at the mercy of a disk latency.
• 8:09
  SPEAKER_00: Because if the number of commits will depend of basically, uh is using f sync, which is the uh system level um disk operation.
• 8:24
  SPEAKER_00: And so if you are doing one message, one commit basically, it means that you need a very very fast uh disk at some point.
• 8:34
  SPEAKER_00: Uh so this is pretty well known, but it also helps to shape uh, I would say, the the use cases where Artemis is pretty interesting, and and ActiveMQ is also very interesting because the report mentioned like a batching on on ActiveMQ actually overperform Artemis in this case.
• 8:58
  SPEAKER_00: So you can see that it's super interesting to see the the thing.
• 9:01
  SPEAKER_00: Um I was surprised by two things to be honest.
• 9:06
  SPEAKER_00: Uh the first one was about IMQP results and rabbit MQ, because initially RabbitMQ used an IMQP plugin and it was considered slow and secondary.
• 9:20
  SPEAKER_00: But since RabbitMQ4, the version bench into the report, the IMQP1.0 support is native in the broker, and so we see that rabbit mq outperforms Artemis and ActiveMQ in this case.
• 9:39
  SPEAKER_00: So it was a surprise because I was I'm not a super great expert in RabbitMQ.
• 9:45
  SPEAKER_00: So in my mind, I say okay, IMQ, IMQP1.0 in Rabbit MQ is just not as good as the others, but actually it's not the case anymore.
• 9:54
  SPEAKER_00: And the report helps to update uh you know the the the state of the art, I would say.
• 10:00
  SPEAKER_00: Um and also the the second thing is uh it was not really a huge surprise, but uh uh I I was happy to see that is all uh about the request reply in Rabbit MQ GMS, uh where we see that it's not that as great as Artemis or ActiveMQ, which is not a surprise because Rabbit MQ GMS is not, I mean, the first citizen class uh in Rabbit MQ uh compared to the other brokers.
• 10:33
  SPEAKER_00: So just to summarize, um really impressed by the report.
• 10:37
  SPEAKER_00: Um great support to confirm what we we know from the ecosystem, but also a couple of surprises, I would say, and I'm super happy to see that.
• 10:49
  SPEAKER_01: Thank you, JB.
• 10:51
  SPEAKER_01: Dan, you know, what was your initial reaction?
• 10:53
  SPEAKER_02: Sure, yeah, definitely um hard to follow up.
• 10:57
  SPEAKER_02: JB just outlined it so well.
• 10:58
  SPEAKER_02: I just I I some so two two notes that I had, it's so funny.
• 11:03
  SPEAKER_02: It's really we're gonna talk about the same things.
• 11:05
  SPEAKER_02: The two notes that I had taken away from it.
• 11:07
  SPEAKER_02: One was the first of all, and this is really just me kicking myself for from my previous time, is is how I missed how Active MQ handles batching so well.
• 11:16
  SPEAKER_02: I think previously I was only doing, you know, I was maybe sending in for the 2022 report, maybe sending, you know, 100,000 messages a time, but I can't believe I was properly batching them.
• 11:26
  SPEAKER_02: That would explain why, as JB noted, why the the throughput that you see is really just a disk limit.
• 11:32
  SPEAKER_02: That's all you're just seeing.
• 11:33
  SPEAKER_02: That's the number, it's the true number that you're getting.
• 11:36
  SPEAKER_02: So I thought by actually, you know, introducing batching, and again, this is how ActiveMQ uh would prefer as their doc state.
• 11:43
  SPEAKER_02: And again, that's why I'm kicking myself because the doc stated very clearly if you're handling these large throughputs, please use batching.
• 11:49
  SPEAKER_02: That's how you get your best performance.
• 11:50
  SPEAKER_02: And I it was not so much as a surprise, but more of a surprise that geez, I can't believe I missed that before.
• 11:56
  SPEAKER_02: Um, the other part was also the the I took it as a note for the penalty of using GMS client on rabbit mq, right?
• 12:04
  SPEAKER_02: Where we see that kind of latency in in the request and reply.
• 12:08
  SPEAKER_02: Now I do have a note on this.
• 12:11
  SPEAKER_02: Uh, I think it it's uh maybe we'll come up later on, but I can definitely explain a bit more on this regard.
• 12:18
  SPEAKER_02: Um, I think it might have had to do a bit with my configuration and specifically with how RabbitMQ's JMS client uh handles basic gets.
• 12:26
  SPEAKER_02: I think there might have been a polling tweak there that I may have missed.
• 12:30
  SPEAKER_02: Um maybe something for a revision moving forward.
• 12:33
  SPEAKER_02: But overall, those are kind of the the key things that I saw.
• 12:37
  SPEAKER_02: Nothing really, I was kind of expecting again, just by nature, that um Artemis would do fine enough.
• 12:45
  SPEAKER_02: I believe that was if we go back to 2022, it was kind of the the field is relatively the same.
• 12:51
  SPEAKER_02: Obviously, we've seen some improvements in rabbit and some capacities, which is also uh nice to see.
• 12:58
  SPEAKER_02: But in general, I'm I'm just happy to see that every kind of uh every technology here kind of has its has its way, kind of has its play.
• 13:08
  SPEAKER_02: Uh there's there's not one that you can read this report and then say, okay, I'm definitely not going with that one.
• 13:12
  SPEAKER_02: I think they all present some pretty valid use cases, again, depending on what you're trying to achieve, what your business, what your enterprise is trying to achieve.
• 13:20
  SPEAKER_01: It all comes down to the use case.
• 13:22
  SPEAKER_01: But I think one thing that's really important before we start you know comparing numbers is understanding what actually counted as a valid result because raw throughput is not practically or particularly meaningful if the conditions underneath it don't reflect the reliability requirements of the workload.
• 13:38
  SPEAKER_01: So, GP, I have a question for you.
• 13:40
  SPEAKER_01: The the whole report is built around this idea of guaranteed delivery, and uh, and it's a hard constraint rather than something you can just measure alongside performance.
• 13:49
  SPEAKER_01: Tell us what that actually you know means in practice.
• 13:53
  SPEAKER_01: What what had to be true in the configuration and and in the test harness for the results to even count?
• 14:01
  SPEAKER_00: Yeah, uh thanks, Mark.
• 14:03
  SPEAKER_00: Uh Mike, and I think that's that's very important because I guess that's the expectation from any user on any system like uh RTMI SACTMQ or RabbitMQ.
• 14:17
  SPEAKER_00: Um the meaning of granted delivery in that case is basically if a message, a person message could not survive an expected power outage or or network failure or whatever, it's a huge issue.
• 14:33
  SPEAKER_00: So that's basically the meaning.
• 14:35
  SPEAKER_00: We have to be able to store the message, rock solid, and deliver again when we can.
• 14:42
  SPEAKER_00: And in practice, to implement that and but the the the ecosystem, we we have very different implementation, but if you take a look at high level, it's basically at two different uh components the persistency, so the way the brokers are basically storing the the messages on some somewhere.
• 15:08
  SPEAKER_00: Most of the time it's it's basically the disk, but it could be somewhere, I would say to be generic.
• 15:14
  SPEAKER_00: But also is how do we deal with harness and reconciliation?
• 15:20
  SPEAKER_00: Meaning that I reload the message from the disk, what does it mean?
• 15:24
  SPEAKER_00: If I have a duplicate, for instance, for any reason, how do I deal with that?
• 15:29
  SPEAKER_00: And it's super interesting in the in the report because we see the different approaches uh of the different brokers, and and it's very interesting, in my opinion.
• 15:40
  SPEAKER_00: So on the persistency uh layer, in Active MQ, we use mostly the uh disk set sync.
• 15:52
  SPEAKER_00: So it means that as a producer, I send a message to the broker, I will get back the acknowledge from ActiveMQ only if I have a guarantee that the message has been persisted into the uh on the disk.
• 16:06
  SPEAKER_00: Uh and we are sure we are not losing any any data.
• 16:10
  SPEAKER_00: On Artemis, it's a little so it also uses the uh the disk, but it's it's a different approach because it tried to leverage the low-level implement um APIs from the disk.
• 16:25
  SPEAKER_00: For by default, if you run on Linux, for instance, you can use async.io, which is a library provided by Linux raw level, to be to be able to deal with four-safe synchronization on the disk.
• 16:40
  SPEAKER_00: Rabbit MQ is basically pretty similar in terms of the approach to our Active MQ using a basic um persistency, but synchronizing the publishers or the producers uh asynchronously, I would say.
• 16:58
  SPEAKER_00: So you can see that at the end of the day, the three brokers behave almost the same way, but using slightly different implementation.
• 17:08
  SPEAKER_00: And for reconciliation, I think it's uh also a different story.
• 17:12
  SPEAKER_00: Active MQ is probably the easiest because we have a sync between the arc knowledge and the state of the disk.
• 17:20
  SPEAKER_00: Uh, there are pros and cons doing that, but it's a huge security, like we have the delivery guaranteed.
• 17:28
  SPEAKER_00: For the others, like RTMIS and Rabbit MQ, they they want to boost performances using different reconciliation discipline, I would say.
• 17:38
  SPEAKER_00: And and here um we have different approaches.
• 17:42
  SPEAKER_00: All what we can see in the report, um, it's the free brokers are pretty rock solid uh in in terms of uh guarantee delivery.
• 17:54
  SPEAKER_00: But it's super interesting.
• 17:55
  SPEAKER_00: I mean, I'm an engineer, so I'm super interested to see the way it's implemented.
• 17:59
  SPEAKER_00: Obviously, I know mostly ActiveMQ and Optimist because I'm working on them, but but it's yeah, also very interesting to see Rabbit MQ in this space.
• 18:08
  SPEAKER_01: Yeah, I think there, you know, another variable uh turns out to matter quite a bit is how you're actually talking to the broker.
• 18:14
  SPEAKER_01: The same broker like can behave very differently depending on the protocol and the client path that you're using.
• 18:20
  SPEAKER_01: So, Dan, you've tested each broker across three different wire protocol tracks.
• 18:25
  SPEAKER_01: JMS, each broker's need a protocol, the AMQP.
• 18:28
  SPEAKER_01: Walk us through why that three-track structure made sense and what it let you see, rather, um, that a single protocol comparison wouldn't have.
• 18:38
  SPEAKER_02: Sure, Mike.
• 18:39
  SPEAKER_02: Yeah, so the good great question here.
• 18:40
  SPEAKER_02: The so this was a decision that was mostly made again in the name of parity uh in keeping it fair.
• 18:46
  SPEAKER_02: Uh, while we could have decided on a singular track or protocol, uh, we really wanted to devise a methodology that made the playing field a bit fairer.
• 18:58
  SPEAKER_02: Uh, since you know, we're comparing similar, similar, albeit different enough technologies, as we've noted already during the session.
• 19:06
  SPEAKER_02: Um, now for what it's worth, the first track and I use track and error quotes here uh decided upon was just the JMS specification.
• 19:15
  SPEAKER_02: So the reason for this was that the JMS spec messaging is something that we find multiple enterprises working with to this day in business critical workflows.
• 19:24
  SPEAKER_02: It's a non-starter.
• 19:26
  SPEAKER_02: Um, every technology within the report here has access to the JMS spec, but they all handle it a bit differently as we see.
• 19:33
  SPEAKER_02: The native protocol in the AMQP 1.0 additional tracks were kind of offshoots of this JMS track.
• 19:41
  SPEAKER_02: Again, because not every technology is handling JMS the same.
• 19:44
  SPEAKER_02: And if you continue down that path, you see some like RabbitMQ, who do support that JMS client, uh, would maybe prefer to handle that communication uh through differently.
• 19:54
  SPEAKER_02: And and we thought it was fair to include those findings, right?
• 19:58
  SPEAKER_02: I think it's fair that if someone is interested in RabbitMQ, well, maybe they're not interested in JMS at all.
• 20:05
  SPEAKER_02: Maybe they're gonna keep it completely native to Rabbit and just talk on that, if they're using Rabbit 4, um, talk on that AMQ 1.0, or if they're using a previous version of AMQ9, whatever the the subversion is there.
• 20:18
  SPEAKER_02: Uh but again to to summarize it kind of in one statement, it was the idea of parity and also the evolution of actually benchmarking the the first just JMS.
• 20:30
  SPEAKER_02: Because I I noticed the differences in performance, so I thought it would be well, let's they don't want to use JMS, or maybe they're not fully comfortable with JMS.
• 20:38
  SPEAKER_02: Let's let's test some other stuff, or maybe they don't want to use AMQP0, or maybe an enterprise is required to use AMQP0 for uh 1.0 for a different uh upstream or downstream service, and they're they need something in the middle, and here are their options, right?
• 20:54
  SPEAKER_02: Maybe they still want to go with ActiveMQ, who does support this through through other means.
• 20:59
  SPEAKER_02: Again, those things are outlines within the report.
• 21:02
  SPEAKER_01: Thank you, Dan.
• 21:03
  SPEAKER_01: Appreciate that.
• 21:03
  SPEAKER_01: You know, I think that now that we've established the methodology, it's time to dig into the numbers.
• 21:09
  SPEAKER_01: And rather than trying to declare a universal winner, um, I want to look at where each broker seems to behave particularly well, where the trade-offs appear, and what's happening architecturally underneath those results.
• 21:21
  SPEAKER_01: So let's start with Artemis.
• 21:23
  SPEAKER_01: You know, um, Chibi, what did the data show about how it handles small message, high concurrency, JMS workloads, and what's actually happening architecturally that makes it perform that way?
• 21:35
  SPEAKER_00: Yeah, thanks, but yeah, that's basically the the follow up about what we started to discuss earlier.
• 21:42
  SPEAKER_00: Um so I think that Artemis, due to the architecture of Artemis itself, uh we clearly see that uh I don't remember remember exactly the numbers, but it was something like uh seven.
• 22:00
  SPEAKER_00: thousand operation per second or something it it was huge on small messages uh so um i think it's mostly due to two things first is the design the architectural design internally to be asking a your journal because you do that that's a similar approach to if you want to Kafka for instance where you are pending records on a shared memory buffer and at the end so you don't have uh all the challenge to rewrite any files or something you just happen happen happen and again and and so that's why it's super fast to add a small messages at the end of the persistent store you're just moving forward and and more specifically when you when you are using async IO so you are leveraging basically the low system level of Linux it's super I mean the the number of threads you can run in parallel it's huge so that explains why Artemis is so good to add small messages on the store that's the first thing but the second thing is in terms of architecture Artemis also use uh an interesting approach for the for the GMS uh wiring so between the client and the broker and that's basically a theme gms wrapper and so every time you are basically communicating is very light for both the client but more interesting interestingly for the broker uh because anytime you create a connection basically it's a thread into the Artemis broker so using this small layer and the kind of facade directly on top of a core engine the Artemis core engine it boosts basically the performances of again small messages because smaller the messages is faster it's gonna be to happen into the the journal that's easy as that so that explain why Artemis is so great for small messages.
• 24:12
  SPEAKER_00: Yeah I think Artemis gives us a great example of how architecture can influence performance under high concurrency but classic it tells a slightly different story especially when durability and batching enter the picture um so again JB Classic shows an interesting split a hard ceiling around 400 commits per second in the strictest durability setting but throughput that climbs dramatically once you introduce batching up to nearly 35000 messages per second walk us through that finding and what it tells about the difference between commit and a message in the context in in this context yeah so that that's also the architectural differences between Artemis and ActiveMQ and ActiveMQ if we're using a batch of one message I mean you maybe hit pretty quickly the limit of your disk because basically it's a one and one between the message and and the disk and we are using KahaDB to store on the disk.
• 25:12
  SPEAKER_00: So we this is a journal file system based database and so basically it means that every commit is executed using an F sync call on the file system.
• 25:26
  SPEAKER_00: And so this by default and and that's by design on the file system this F-Sync function don't paralyze across storage queues so it means that if you have a lot of concurrent uh production you are basically uh you know reducing the flup because you have contention on the file system due to the f-sync so in the report we can see that we probably hit the physical limit of a file system around I don't remember I think it was around 500 uh operation per second or something like that so second or something like that so it was so it means basically uh 5500 commits per second and so that's very uh small compared to RTMEs and it's where then when you start to batch in active nq you are basically grouping a set of messages in one GMS transaction and so here that's the opposite of RTMEs you have a huge boost of performances because you have only one IU operation to do to store a bunch of messages and so that's why our active MQ is actually better than RTMEs when you have a batch of messages and I I I I think that the report mentioned as soon as uh Dan started to use batching we got something like uh correct me if I'm wrong uh um Dan but I think it's 35000 messages per second or something so which is yeah exactly so which is way better than uh the other two brokers so my my my view and my takeaway from that is by default depending on your GMS usage you may not be you may not have exactly what you expect from the broker but if you are if you know exactly how to use active mq efficiently using a batching of messages on a single GMS transaction etc you are basically changing completely the the the bottleneck that you had before so my point is and it's some it's something that I say often to the to to the users is the broker is one thing but the logic implemented by the clients is super important for for active mq don't expect to do everything on the broker you also have to implement your applications to leverage and to use the JMS client for active mq the most efficient way you can regard active mq yeah I think that the distinction between commits per second and messages per second is really important because those numbers can look contradictory until you understand what the broker is actually doing and the logic that's driving it.
• 28:33
  SPEAKER_01: But I think Mesha you know um rabbitmq rather gives us another example of where the implementation path makes an enormous difference.
• 28:40
  SPEAKER_01: So Dan RabbitMQ has had one of the more striking numbers in the report you know request reply latency that's around 105 milliseconds over the JMS shim but it drops to about 1.2 milliseconds over the uh on the on the native client tell us the story behind that number and what's exactly um what's actually driving that that gap sure mike uh thanks for the question yeah so the the 105 ms delay uh certainly is a bit jarring especially compared to all the numbers uh in this report but i think after some further inspection can certainly be explained in and with a with a good note if anything so uh I believe the rabbit mqms library itself implements a times receive method call which just acts as a basic get which pulls uh which I mentioned before it pulls with a fixed 100 millisecond backoff so each time that a message would go round trip it's kind of subject to this polling interval this also would explain why the number doesn't get impacted when you change the payload size or when you change how many consumers and how many producers are on it.
• 29:45
  SPEAKER_02: They're all just subject to that one polling thing very similar to how we saw the upper limit with the dB rate limit for active MQ.
• 29:52
  SPEAKER_02: Similar in the sense that it's hardcapped not by actual performance by but by some other uh thing kind of impacting it.
• 30:00
  SPEAKER_01: So when you use the native client as noted uh as the push consumer we see the reply arrive almost as soon as it's routed so the real cost is about 1.2 milliseconds still confirming to that guaranteed uh delivery but again I think this was just a slight uh misconfiguration for the rabbit mqms library specifically if anything right I think the result really illustrates that you know why saying rabbit mq latency is X isn't particularly useful without also asking which client and which protocol and the same principle shows up when you move from you know point to point messaging to fan out.
• 30:36
  SPEAKER_01: So JB pub sub fan out scaled very differently across your brokers as you add in more subscribers.
• 30:43
  SPEAKER_01: Talk us through what you saw when there went um what you saw there and and and what it reveals about each broker's underlying you know underlying architecture.
• 30:53
  SPEAKER_01: Yeah but that's an interesting part of a repo actually uh and we can see huge differences uh between the brokers uh and I think we have to take a look beyond beyond the hood you know what's going on in the in the broker that can explain these uh differences uh let's start with Artemis I think that the main reason is the the the the address and queue model used into Artemis so basically Artemis use address and queue which is completely decoupled uh it gives a lot of flexibility but basically it means when a message is published the payload is stored once and the broker manages lightweight delivery pointers on top of a uh of these storage to deliver to the subscribers uh so that that's the way that it's basically decoupling uh address and queues which is in my opinion a a nice design uh and it's the way that it's implemented uh active mq manage durable subscription uh directly in the car db directly on the storage and that's why I think that ActiveMQ scale the best in regular JMS mode because basically uh you you you attach I would say the uh pub sub to the car db uh or to the storage another feature which is super interesting in ActiveMq is the virtual topics where basically you can also boost so by default when you you when you use a pub sub um so basically topics in JMS you are publishing once but you can have multiple subscribers on the topic and so basically you have to uh deliver the the same message to the different subscribers and so it means that a single subscriber if you have a bunch of messages in a topic cannot really concurrently consume messages uh and that's the limitation in the JMS peg and so what mq does is basically you can introduce virtual topics that basically create distinct physical cues per subscriber and so you have a boost of performances because in that case you can have concurrent consumption on on the on the topic so we which is super super interesting and and that explains why active mq is so uh so good in terms of pure pub sub in gms mode rabbit mq now uh it's more broker side challenge more than a client challenge because in rabbit mq each durable subscriber is mapped to an independent queue exactly the same approach as virtual topic but it means also that each running its own earl and after process inside the uh disk sync and so the publisher confirm he has to wait until opsubs queues at first to disk and without basically a cross-queue group commit the publisher looks like very slow because he has to wait everyone uh and every single subscriber before basically moving forward which is a little bit different so that explains basically the differences between the the free brokers thank you jb i think again up to this point we've talked quite a bit about broker specific behavior but one of the more interesting parts of this report is what happens when all three brokers are asked to communicate through the same standardized protocol amq 1.0 specifically but um that's where some of these assumptions from jms results start to change so um let's let's start with you again jb uh track c the amq uh amq 1.0 track let it seemed to flip the leaderboard in some interesting ways compared to the jms track walk us through what changed and why amqp 1.0 performance didn't just mirror what you saw on track a yeah so that that's interesting because amqp is a very in term of uh design and behavior is very different from jms per se i would say so rabbit mq is really so let me take a step back in in track a we we we saw that rtimes dominated the small messages battle i would say and and for reasons that we explained before in track c obviously rap rabbit mq won every single uh batch equal one and the number of messages um and on the other hand if you we take a look on active mq on track a batching uh to basically active mq to a huge number of messages processed on track c stay uh flat between the number of messages i don't remember the number but it was between 400 messages per second and 500 messages per second so it was basically flat whatever the the batch uh what was defined 100 000 it doesn't change much um and why is because by default in IMQP it's a non-transactional approach so track C measure the non-transaction IMQP 1.0 approach where basically durability so storing into the uh the file system is enforced per message not per batch so because remember the batch works when we have this uh a unique JMS transaction so we basically groups we have a group of messages in one JMS transaction in track C we tested per message settlement and so that's why classic does not support this kind of uh group commit in MQP and and that explained why Artemis has this kind of uh flat number which is the the performances of the disk again uh so I think that potentially is a good improvement in in active mq to implement I don't know maybe um new group commits in MQP stuff like that.
• 37:34
  SPEAKER_00: Rabbit MQ on the other hand is really um qp at core of a broker and so they can do per message confirm after persist map and so that's a huge boost of per performances.
• 37:50
  SPEAKER_00: So that that's the reason why we see this difference between rabbit mq and and active mq and why in active mq whatever the size of a batch it doesn't really matter in terms of performances it's basically flat.
• 38:06
  SPEAKER_02: This is specific to MQP uh I'm not sure that today IMQP is so for rabbit mq it's it's a strong use case I don't see that the there's a lot of users on IMQP for active mq to be honest thank you jb appreciate that uh Dan let's let's jump over to to badging right uh active mq's behavior under amq was almost the mirror image of its behavior on the JMS track badging helps enormously in one and does almost nothing in the other right so can you narrate what's happening under the hood there sure absolutely thanks Mike yep so um you know it's uh I'm very proud to say it but I I mean JB pretty much just outlined the whole thing what's happening under the hood I'll just quickly mention he he explained it much better than I ever could but essentially the track C the MQP 1.0 as JB noted rabbit mq doesn't do transactions for this so I saw it appropriate to not have transactions for the others as well this means that active mq in this track is not working under transactions which would explain kind of almost everything that JB was just talking about.
• 39:16
  SPEAKER_02: It's kind of subject to those limits and such and that's why we don't see kind of the same kind of batching result.
• 39:22
  SPEAKER_02: Why?
• 39:22
  SPEAKER_02: Because it's uh I believe being bottlenecked by something else in this capacity now.
• 39:27
  SPEAKER_01: Fair fair so let's shift away then from performance rankings for the moment uh and let's talk about the research process itself one thing I appreciate in the report is that it doesn't pretend benchmarking is perfectly clean.
• 39:39
  SPEAKER_01: There were results that you know there were results that forced the team to go back to investigate to rerun the test and in some cases remove the data entirely.
• 39:48
  SPEAKER_01: That's important because a benchmark is only as useful as the methodology behind it.
• 39:52
  SPEAKER_01: JB thank you guys so much um and to everyone listening stick around for part two because it's gonna get really more exciting a lot more you know conversation to be had here uh I think there's a lot of takeaways today but we are gonna wrap today and bring around JB and Dan for round two.
• 40:08
  SPEAKER_01: Uh so really appreciate y'all tuning in for this episode of What's the hoot make sure you're subscribed for the next one and I will see you uh all around the Ben and I appreciate your time.
• 40:17
  SPEAKER_01: Have a great day