Latest / 1 More Paper / Adopting Large Language Models to Automated System Integration — Pesl (2025)
Transcript
- You’re listening to “Adopting Large Language Models to Automated System Integration,” by R.D. Pesl. Published in 2025. Abstract. Adopting Large Language Models to Automated System Integration Robin D. Pesl University of Stuttgart, Stuttgart, Germany the email address Abstract Modern enterprise computing systems integrate numerous subsystems to resolve a common task by yielding emergent behavior. A widespread approach is using services implemented with Web technologies like REST or OpenAPI, which offer an interaction mechanism and service documentation standard, respectively. Each service represents a specific business functionality, allowing encapsulation and easier maintenance. Despite the reduced maintenance costs on an individual service level, increased integration complexity arises. Consequently, automated service composition approaches have arisen to mitigate this issue. Nevertheless, these approaches have not achieved high acceptance in practice due to their reliance on complex formal modeling. Within this Ph.D. thesis, we analyze the application of Large Language Models (LLMs) to automatically integrate the services based on a natural language input. The result is a reusable service composition, e.g., as program code. While not always generating entirely correct results, the result can still be helpful by providing integration engineers with a close approximation of a suitable solution, which requires little effort to become operational. Our research involves (i) introducing a software architecture for automated service composition using LLMs, (ii) analyzing Retrieval Augmented Generation (RAG) for service discovery, (iii) proposing a novel natural language query-based benchmark for service discovery, and (iv) extending the benchmark to complete service composition scenarios. We have presented our software architecture as Compositio Prompto, the analysis of RAG for service discovery, and submitted a proposal for the service discovery benchmark. Open topics are primarily the extension of the service discovery benchmark to service composition scenarios and the improvements of the service composition generation, e.g., using fine-tuning or LLM agents. Introduction. Automated service composition describes the emergence of combining multiple services to a composite service. Automating this process yields the advantages of reduced manual effort, faster time-to-market, and agile adoption to changed business needs, resulting in an overall strategic benefit for the company. An example would be an automotive vendor wanting to integrate roadside services like parking spot booking. An automated approach allows for the integration of services that are not available during design time without further manual development effort. Previous approaches to automated service composition rely on formal models, always producing correct results while requiring extensive manual modeling. With the advent of Large Language Models (LLMs), it has become feasible to process natural language queries and semi-structured documentation automatically, i.e., formal and natural language parts. Employing LLMs for automated service composition could mitigate the issue of complex formal modeling by allowing developers to express their requirements in natural language while generating a code recommendation fully automatically. This leads to our overarching research question: How well can LLMs be employed for automated service composition? The remainder of the paper is structured as follows. In Section 2, we give an overview of the current literature regarding service composition in Information System Engineering (ISE) and Service-Oriented Computing (SOC) and the application of LLMs for service compositions. Then, we state our research methodology in Section 3. In Section 4, we clarify our contributions. Section 5 shows what we already achieved. We elaborate on our planned work in Section 6 and conclude with Section 7. 2 State of the Art We provide a short literature overview to motivate the topic’s relevancy and explain the current state of the art. This includes a brief description of classical service composition approaches, the subfield of service discovery, its relevance for ISE, and initial ideas to apply LLMs in SOC. 2.1 Service Composition Automated service composition has been a field of research in ISE and SOC for more than two decades. While ISE mainly focuses on its positioning in automating parts of the requirement engineering process to reduce workload and decrease time-to-market, SOC concentrates on its technical implementation. This contains aspects like component access, conversation management, control flow, dataflow, and data transformation. While there was initially high creativity in creating solution approaches, the research slowed down. Nevertheless, there is still a lack of a comprehensive, viable solution. Famous classical approaches rely on AI planning, which computes a plan, i.e., a sequence, of service invocations based on formal modeling of the service and the composition requirements. These approaches can be domain-specific or domain-independent. Further approaches rely on finite state automata to model the service interaction known as the “Roman model”. While always producing correct results, these classical approaches require laborious and erroneous formal modeling, leading to brittle solutions and low application in practice. In contrast, services are often documented using a semi-structured OpenAPI specification in JSON or YAML. It consists of general information about the service, like name or host, and the endpoints, i.e., the APIs that offer the actual functionality. The endpoint specification again contains natural language elements like a description and structured elements like input and output schemas. Our approach relies on OpenAPI specifications for implementations as these are the state of practice. A subfield of automated service composition is service discovery, which identi-fies relevant services within a potentially vast set of all available services. Initial ideas concentrate on centralized registries across vendors implement, e.g., in the Universal Description, Discovery, and Integration (UDDI) specification. These share the same drawbacks of requiring extensive laborious manual modeling and opposing workload reduction efforts. More recent approaches try to leverage already present OpenAPI documentation. In our work, we analyze the application of Retrieval Augmented Generation (RAG) with OpenAPI to realize service discovery to allow automated natural language processing, sidestepping any manual effort. The relevancy of service composition and service discovery is backed by a long list of literature in the ISE community, e.g.. It enables steering system design, streamlining development, dynamic system changes, reduced manual labor, increased scalability, avoids human-induced errors, and allows agil reactions to changed business needs. Often, it is considered as part of the requirement engineering process, e.g., using ontologies or formal models. Domains include Smart Cities or cloud computing. Recent implementations also support OpenAPI specifications. Our work contributes to this knowledge corpus by analyzing how LLMs can be applied to the problem. 2.2 LLMs for Automated Service Composition LLMs achieve remarkable results in natural language understanding, processing, and generation. Initial ideas adopt an encoder-decoder architecture. Newer approaches use a decoder-only approach for text generation and encoder-only (embedding) models for similarity computation. Within SOC, we proposed initial concepts of using LLMs for service composition. These still face the issues of input token limitations, imperfect results, and hallucinations. Another approach to integrating LLMs with services (tools) is LLM agents. These incorporate tool invocations into the chat interaction. While facing similar problems like tool/service selection, the main difference to service composition is that the result of service composition is an executable, reusable artifact, e.g., as code. In contrast, LLM agents invoke the tools directly during the answer creation. A significant problem is the lack of appropriate benchmarks. Although some initial proposals like RestBench exist, a general benchmark across numerous domains is still missing. We add to this by introducing generalized benchmarks. 3 Research Method Figure 1 shows our research methodology from left to right. The first item is a general software architecture, i.e., a method to employ LLMs for automated service compositions. Using this architecture, we can implement a prototype and measure the performance and implications of the approach. Next, we look into the literature and analyze how to classify our method in the existing taxonomy of Lemos, Daniel, and Benatallah and how the taxonomy needs to be extended. In parallel, we examine service documentation chunking, i.e., extracting the most relevant parts to allow using RAG for service discovery. This allows us to mitigate prompt input token limitations. We evaluate the RAG chunking approaches first by employing the exiting RestBench benchmark, then by introducing our custom SOCBench-D benchmark generalized across all domains of the Global Industry Classification Standard (GICS). Finally, we bridge the gap between service discovery and our software architecture by creating a benchmark comprising services based on the GICS domains, measuring the end-to-end performance from prompt to final service composition. Further experiments could contain user studies to measure the time-saving of employing LLMs versus manual labor or applying LLM agents to improve reasoning. 4 Contributions Following our research methodology from Figure 1, we introduce four contributions. The contributions are as follows: 1. The Compositio Prompto software architecture. 2. Our extension to Lemos’ taxonomy, which results in an extended taxonomy. 3. The service discovery using RAG. This comprises the analysis of RAG using the existing RestBench and the creation of the SOCBench-D benchmark. The resulting artifacts are a query-based service discovery benchmark and an algorithm that can dynamically create such a benchmark. 4. The SOCBench-SC service composition benchmark. It contains the analysis of current LLMs, the benchmark itself, and the benchmark creation algorithm. 5 Preliminary Results We already worked on the method, the taxonomy, and the service discovery with RAG. Open points are in the full service composition and subsequent studies. 5.1 Compositio Prompto First, we introduce our architecture “Compositio Prompto” to employ LLMs for automated service composition shown in Figure 2. It uses a task, service documentation, and an in- and output schema as input to create a prompt. The prompt is then fed into the LLM, creating an executable service composition. To evaluate Compositio Prompto, we implemented a fully operation prototype and performed a case study from the automotive domain. Our prototype uses a natural language query as a task, OpenAPI specifications as service documentation, JSON schemas as in- and output schemas, and Python code for the executable service composition. The results show that currently, only large models with more than 70B parameters can solve at least some tasks perfectly. Nevertheless, even small models like Llama 3 8B produce close approximations of our manually crated sample solution. Therefore, we conclude that current LLMs can be used to create a code recommendation that only needs little adaption, i.e., working time, to become operational. Further research is needed to achieve full automation. 5.2 Extended Taxonomy To analyze whether service composition with LLMs can already be expressed using the well-known taxonomy of Lemos, Daniel, and Benatallah, we examine for each category if it already covers the LLM capabilities adequately. The result is an extended taxonomy, which comprises the additional subcategories needed to include the LLM capabilities in natural language understanding, semantic processing, and text generation (submitted). We validated our taxonomy extension using four existing LLM-based approaches for service composition from literature. They show that each introduced new subcategory is indeed necessary. Limitations are primarily in the original methodology, which only includes taxonomy elements that stem from actual service composition approaches. Further extensions may be necessary once novel service composition approaches arise. 5.3 OpenAPI RAG The Compositio Prompto experiments highlight the challenge of the limited input token length of LLMs. It leads to only being able to input only parts of the service documentation. An example is that the OpenAPI of Spotify alone does not fit into the context size of OpenAI’s GPT4o, leaving alone inputting additional services. A technique to mitigate this issue is RAG, which splits the input data into smaller chunks. The RAG system performs a semantic search using an embedding model based on these chunks and inserts the most relevant chunks into the prompt. The benefit is that the inserted chunks are much smaller than the complete input data while revealing only the most relevant information. We apply RAG to service discovery to determine the influence of the model and chunking strategy (to appear). We rely on the RestBench benchmark, which consists of the Spotify and TMDB OpenAPI specification and pairs of natural queries with expected endpoints. Our results show that it is beneficial to split the OpenAPI by endpoints. Nonetheless, the lack of general benchmarks arises. Therefore, we extended our evaluation further by introducing the novel service discovery benchmark SOCBench-D, which generalizes across the GICS domains (submitted). We execute it using OpenAI’s text-embedding-3-large, Nvidia’s NV-Embed-v2, and BGE’s bge-small-en-v1.5. Our results show that the choice of the chunking strategy is insignificant across all domains. The number of received chunks is particularly relevant to the overall performance. The second factor is the embedding model. The Nvidia model outperforms the OpenAI model, which outperforms the BGE model. Nonetheless, the BGE model reveals reasonable results. In practice, this leads to the consideration that when resources are sparse, the BGE model can be used; when familiar with the OpenAI tooling, the OpenAI model can be used; and when interested in the best results, the Nvidia model can be employed. 6 Future Work Our next research effort will comprise the analysis of complete service compositions incorporating the RAG-based service discovery. The basic idea is to use a query-based service discovery benchmark like RestBench or SOCBench-D and extend it to code analysis. This can be done by static code analysis, unit test-like testing, or creating custom mock instances of the services and tracking invocations. The optimal approach has still to be determined. The result is a benchmark for natural language service composition approaches, generalized across the GICS domains. Further, we want to measure actual implications on development time savings, sustainability aspects, and advanced approaches like LLM agents. These allow logical reasoning, which may reduce hallucinations and improve the composition quality. 7 Concluding Remarks With this Ph.D. thesis, we want to analyze the potential of employing LLMs for the well-known yet unresolved problem of automated service composition. We introduced the Compostio Prompto architecture to realize automated service composition with LLMs in practice. Further, we examined the usage of RAG for service discovery first by using the real-world RestBench benchmark and then by our cross-domain SOCBench-D benchmark. Next, we will create a benchmark for general service composition cases and analyze how well current LLMs perform. Other open points are the improvement of the result generation, e.g., by employing LLM agents, and the analysis of applicability in practice, e.g., by performing a user study. Acknowledgments. I want to thank my supervisor, Prof. Dr. Marco Aiello, for his ongoing support throughout my Ph.D. journey. This preprint has not undergone peer review (when applicable) or any post-submission improvements or corrections. The Version of Record of this contribution is published in Intelligent Information Systems. CAiSE 2025. Lecture Notes in Business Information Processing, vol 557. Springer, Cham., and is available online at the linked source. Disclosure of Interests. The author is listed as inventors of a patent, which covers Compositio Prompto for the automotive domain. 1. Apostolakis, I., Mainas, N., Petrakis, E.G.: Simple querying service for OpenAPI. descriptions with semantic extensions. Information Systems 117, 102241 (2023) 2. Ben-Sassi, N., et al.: Service discovery and composition in smart cities. In: Information Systems in the Big Data Era. pp. 39–48. Springer (2018) 3. Berardi, D., et al.: Automatic Composition of E-services That Export Their Behavior. In: ICSOC. pp. 43–58 (2003) 4. Bianchini, D., et al.: Ontology-based methodology for e-service discovery. Informa- 5. Curbera, F., et al.: Unraveling the web services web: an introduction to SOAP. WSDL, and UDDI. IEEE Internet Computing 6, 86–93 (2002) 6. De Giacomo, G., Patrizi, F., Sardiña, S.: Automatic behavior composition synthesis. 7. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep. bidirectional transformers for language understanding. In: NAACL-HLT 2019. pp. 4171–4186 (2019) 8. Dourdas, N., et al.: Discovering remote software services that satisfy requirements:. Patterns for query reformulation. In: CAiSE. pp. 239–254. Springer Berlin Heidel-berg, Berlin, Heidelberg (2006). the linked source 9. Jerbi, I., et al.: Request relaxation based-on provider constraints for a capability-. based naas services discovery. In: CAiSE. pp. 611–627. Springer (2023) 10. Lemos, A.L., Daniel, F., Benatallah, B.: Web service composition: A survey of. techniques and tools. ACM Comput. Surv. 48 (dec 2015) 11. Lewis, P., et al.: Retrieval-augmented generation for knowledge-intensive NLP tasks. 12. McDermott, D.V.: Estimated-Regression Planning for Interactions with Web Ser-. vices. In: AIPS. pp. 204–211. AAAI (2002) 13. Mehandjiev, N., Lécué, F., Carpenter, M., Rabhi, F.A.: Cooperative service compo-. sition. In: CAiSE. pp. 111–126. Springer (2012) 14. Mialon, G., et al.: Augmented language models: a survey (2023), the linked source. org/abs/2302.07842 15. MSCI Inc., Standard & Poor’s: Global industry classification standard (GICS). the linked source 16. OpenAI: Function calling and other API updates, the linked source. index/function-calling-and-other-api-updates/, last accessed 2024-07-18 17. OpenAPI Initiative: OpenAPI specification. the linked source (2021). version 3.1.0, last accessed 2025-03-15 18. Oriol, X., Teniente, E.: An ontology-based framework for describing discoverable. data services. In: CAiSE. pp. 220–235. Springer, Cham (2018) 19. Pesl, R.D., Aiello, M.: Revisiting Lemos’ taxonomy for service compositions with. large language models. In: ICWS (2025), submitted 20. Pesl, R.D., Klein, K., Aiello, M.: Verfahren zur Nutzung von unbekannten neuen. 21. Pesl, R.D., Mathew, J.G., Mecella, M., Aiello, M.: Advanced system integration:. Analyzing OpenAPI chunking for retrieval-augmented generation. In: CAiSE (2025), the linked source, to appear 22. Pesl, R.D., Mathew, J.G., Mecella, M., Aiello, M.: Retrieval-augmented generation. for service discovery: Chunking strategies and benchmarking. TSC (2025), submitted 23. Pesl, R.D., Stötzner, M., Georgievski, I., Aiello, M.: Uncovering LLMs for service-composition: Challenges and opportunities. In: ICSOC 2023 WS. Springer (2024) 24. Pesl, R.D., et al.: Compositio Prompto: An architecture to employ large language. models in automated service computing. In: ICSOC 2024. pp. 276–286. Springer (2025) 25. Radford, A., Wu, J., Amodei, D., Amodei, D., Clark, J., Brundage, M., Sutskever. I.: Better language models and their implications. OpenAI blog 1 (2019), https: //openai.com/index/better-language-models/, last accessed 2024-11-28 26. Radford, A., et al.: Improving language understanding by generative pre-training. (2018) 27. Sheshagiri, M., DesJardins, M., Finin, T.: A planner for composing services described. in DAML-S. In: 13th ICAPS WS on planning for Web services (2003) 28. Soki, A.T., Siqueira, F.: Discovery of RESTful Web services based on the OpenAPI. 3.0 standard with semantic annotations. In: AINA. pp. 22–34. Springer (2024) 29. Song, Y., et al.: RestGPT: Connecting large language models with real-world. 30. Wang, H.H., et al.: A formal model of the semantic web service ontology (WSMO). Information Systems 37, 33–60 (2012) 31. Yao, S., et al.: React: Synergizing reasoning and acting in language models (2023), the linked source 32. Zachos, K., et al.: Discovering web services to specify more complete system requirements In: CAiSE. pp. 142–157. Springer (2007)