Latest / 1 More Paper / Insecure by design? A human-centric security perspective on AI-assisted software development — Glas et al. (2026)
Transcript
- You’re listening to “Insecure by design? A human-centric security perspective on AI-assisted software development,” by M. Glas and colleagues. Published in 2026. Contents lists available at ScienceDirect Computers & Security journal homepage: the linked source TC 11 Briefing Papers Magdalena Glas a,∗, Christoph Nirschl a, Bar Lanyado b, Johan van Niekerk c a University of Regensburg Universitätsstr. 31, Regensburg, 93047, Germany b Lasso, Beit Shamai St 10, Tel Aviv, 6701838, Israel c Noroff University College Tordenskjolds gate 9, Kristiansand, 4612, Norway a r t i c l e i n f o a b s t r a c t Generative artificial intelligence (AI) tools are increasingly used in software development, improving the effi-ciency of software developers. However, this adoption introduces notable security challenges. AI/generated code is not secure by default, as it is often based on large-scale training data that includes open-source code of varying quality and trustworthiness. Developers using these tools may be unaware of the associated risks or may place excessive trust in the security of the output. This briefing paper outlines the key security risks associated with generative AI and offers human-centered strategies for mitigation. Since these risks arise not only from how generative AI models are built but also from how humans interact with them, we adopt a human-centric per-spective. To this end, we provide recommendations for individuals, organizations, and educators to help harness the potential of generative AI in software development while effectively managing the associated security risks. 1. Introduction. Developers increasingly use AI assistants like ChatGPT or GitHub Copilot in software development for tasks like generating, debugging, and documenting code. These assistants have great potential to make software development faster, less tiring, and more attainable to novices. By offloading routine coding tasks to generative AI, developers can focus more on higher-level design and problem-solving. Since the AI can turn natural language prompts into executable code, developers may no longer need prior programming knowledge or an understanding of syntax to produce working solutions. However, the potential that AI assistants offer comes with inherent security risks. AI/generated code may include insecure practices or vulnerabilities that users might overlook, especially if they rely too heavily on the tool without critically evaluating its output. Seasoned developers are generally more aware of these risks, as they have the knowledge and experience to spot insecure code or question the suggestions provided by AI tools. In contrast, novices and less experienced programmers might lack the skills or confidence to assess the security implications of AI/generated code. Ultimately, both seasoned developers and beginners need to consider security when using AI assistants for coding. The goal is to enable everyone to benefit from the advantages AI brings to software development while still ensuring that the code they produce is secure. The capabilities of AI coding assistants like ChatGPT are grounded in the technological evolution of transformer-based Large Language Models (LLMs), first introduced by Vaswani et al. in 2017. The transformer architecture replaced earlier sequential models such as LSTMs with a parallelized attention mechanism, allowing models to scale dramatically in both size and performance. OpenAI’s GPT series exemplifies this progression, growing from GPT-1 to the much larger GPT-4 and GPT-4o, with increasing ability to perform few-shot and zero-shot learning. These models are pre-trained on vast corpora of text and code, such as public GitHub repositories, technical documentation, and Q&A forums, allowing them to learn statistical associations between language tokens, including programming constructs. While the current generation of transformer-based LLMs marks a major leap, early AI models, such as Hidden Markov Models, were capable of generating sequential data, although they were limited in scope. Significant progress in AI, particularly in Natural Language Processing (NLP), was marked by the advent of deep learning models like RNNs, LSTMs, and GRUs, enabling the generation of longer text content. Recently, the landscape of software development has been dramatically transformed by the rapid advances in AI models. These models, often based on the attention-based transformer architecture, became widely known and available through pre-trained models like OpenAI’s GPT series, Codex, Google’s PaLM, Meta’s LLaMA, and Anthropic’s Claude. The AI assistants built upon these underlying machine learning models have emerged as powerful tools intended to aid in the software development lifecycle and improve developer productivity. Prominent ex-amples include GitHub Copilot, ChatGPT, Google Gemini, and Microsoft Copilot. To create code, the underlying models rely on large datasets of publicly available code, sourced from platforms like GitHub. They oper-ate by taking text prompts, such as function definitions or natural language descriptions, and generating corresponding code outputs. AI assistants constitute a novel code inclusion path, distinct from traditional mechanisms like importing or forking dependencies. As AI assistants become increasingly integrated into software development workflows, they offer a range of benefits across the entire development process. Developers are using these tools to support a variety of tasks throughout the software development lifecycle, including code generation, testing, debugging, review, and documentation. By automating repetitive and time-consuming activities, AI allows developers to focus more on complex, higher-level challenges such as architecture and design. This shift not only accelerates development timelines, but also contributes to improved code quality and fewer errors. Reflecting these advantages, adoption has become widespread. Recent surveys show that well over 90% of professional developers use AI-assisted tools in their workflows, with the majority recognizing clear benefits, demonstrating a strong and growing level of trust in AI-/generated code. Furthermore, AI assistants lower the entry barrier for novice programmers. As AI assistants can generate code directly from natural language, users do not necessarily need to know the syntax of a programming language. As such, individuals who may lack formal training in programming languages are suddenly able to create functioning code. Beyond code generation, AI assistants also provide contextual suggestions, clarify programming concepts, and flag potential errors, enabling novices to learn as they code and gradually build their skills through guided feedback. Over time, many novices report lasting improvements in their skills, indicating a durable, domain-specific learning effect. This aligns with broader research suggesting that regular use of AI assistants can support learning and skill development in programming contexts. Despite these advantages, this widespread use introduces new concerns, especially regarding security. Using AI assistants in software development carries the risk of creating a misleading sense of competence, especially in inexperienced users, as successful outcomes may reinforce a misleading perception of their own skills. This, in turn, may lead to an uncritical acceptance of AI/generated suggestions and reduce em-phasis on critical quality assurance practices, such as thorough testing and peer code review. Coupled with growing trust in and (over-)reliance on AI assistants, this can lead to unintentionally introducing security vulnerabilities. In the following, this briefing paper outlines key security risks in AI-assisted software development and provides recommendations for individuals, organizations, and educators to address these risks. 2. Security risks of AI-assisted software development. We identified three key problems that contribute to security risks in AI-assisted software development. These are the tendency of AI assistants to generate insecure code, developers’ uncritical use of such code, and the absence of methods or benchmarks to assess its security. The three problems are outlined in the following. Problem 1: AI assistants do not necessarily produce secure code. The pre-trained nature of AI models introduces significant limitations when it comes to the security of the created code. The quality, diver- sity, and representativeness of the training corpus directly affect the model’s performance. This training data, often pulled from open-source platforms like GitHub or StackOverflow, includes code of varying quality, and importantly, it is not curated for security. As a result, these models can learn and reproduce insecure coding patterns, even if the generated code is fully functional and appears correct at first glance. Fig. 1 shows an example of such a vulnerable code when advising an AI assistant to generate code that allows users to upload files to a website. The im-plementation operates correctly. However, from a security standpoint it fails to enforce upload sanitization and restrictions, thereby permitting directory traversal, unintended file overwriting, and the submission of executable (and potentially malicious) files. The security of AI/generated code depends on multiple factors. One key factor is the specific AI model being used. Different models may produce more or less secure code depending on the quality of the training data and the way the model was fine-tuned. Second, there are clear differences between programming languages. Languages like Python or JavaScript, may have better community practices around security, while others, especially hardware-related languages like Verilog, are more prone to insecure output. In some cases, the vulnerabilities are not obvious in the code itself but become critical once the code is integrated into a larger system. Even when the model generates syntactically sound code, the inclusion of external libraries or tools can introduce further vulnerabilities. AI models often suggest the inclusion of packages to solve a given task. However, these suggestions are not always reliable. In some cases, the assistant may hallucinate packages that do not exist at all. In other cases, it might recommend outdated or vulnerable libraries. These insecure dependencies introduce risks that developers may not detect, especially if they trust the AI output without further validation. Overall, while AI tools can speed up coding and help with complex tasks, they do not guarantee secure results. Without proper filtering, validation, and review, AI/generated code can quietly introduce security flaws that are hard to detect later in the development process. Problem 2: Developers often use AI/generated code without full awareness of security risks. Beyond the technical limitations of AI models, a significant risk arises from how developers adopt and use AI-/generated code, often without fully considering potential security implications. This behavior continues a long-standing trend in software development, relying on ready-made code snippets from platforms like StackOverflow without fully understanding their implications. However, the use of AI assistants increases this risk. Because AI assistants provide tailored, context-specific code on demand, they create a sense of efficiency and reliability that can lead users to assume the code is safe. As long as it compiles and pro-duces the expected result, developers often accept it without considering whether it introduces security issues. This risk is especially pronounced among less experienced developers. Many are capable of generating working solutions with the help of AI, but lack the security knowledge to assess whether the code is actually secure. In these cases, functionality becomes the primary in-dicator of quality: if the code works, it is considered “good enough”. Security checks are skipped, not necessarily out of carelessness, but because developers may not realize what to look for or do not know that a vulnerability could be present. Even experienced developers are not immune to this behavior. Al-though they may be aware of secure coding practices, they are still at risk of overtrusting AI/generated code. Prior studies have shown that developers often assume the AI “knows better”, which can lead to mis-placed confidence in its output. Developers per-ceive AI/generated code as more secure or higher quality than human-written code, simply because it comes from a seemingly intelligent system. This perception reduces the likelihood of questioning the results. On top of that, developers also tend to overestimate their ability to review and evaluate AI/generated suggestions. In interviews, developers reported high confidence in their capacity to catch mistakes or security flaws in AI output, yet their actual performance in identifying risks did not match this confidence. This gap between per-ceived and actual oversight contributes to the silent spread of insecure code, even in teams that are otherwise security-aware. In short, the ease and speed of using AI/generated code can over-shadow security concerns, both for novices and experienced developers. Without targeted interventions or changes in development practices, insecure code may continue to be introduced simply because it appears to work. Problem 3: Lack of methods and benchmarks to assess security of AI/generated code. Even if developers become more aware of security concerns, a critical barrier remains: there are few effective methods to systematically evaluate whether AI/generated code is actually secure. Traditional vulnerability handling processes, such as manual code reviews and static analysis, face major challenges when applied to AI-/generated code. These tools and workflows were built around the assumption that code is written by humans and follows common coding patterns. However, AI/generated code can introduce vulnerabilities that are semantically identical to known issues but look syntactically different, making them harder to detect using con-ventional tools. As a result, known weaknesses may go undetected because they evade the pattern-matching logic of traditional tools. In addition to these technical challenges, the volume and speed of AI/generated code can overwhelm manual review processes. Security analysts and reviewers may struggle to keep up with the output of AI tools, especially in fast-paced development environments. This further increases the risk that insecure code slips through without proper validation. Another limitation lies in the evaluation itself. While there are many benchmarks for assessing the performance of AI/generated code, there are few widely accepted benchmarks focused specifically on security. Some research efforts have proposed new ways to test AI tools for vulnerability introduction, but these security-focused benchmarks have not yet seen broad adoption within the AI and machine learning communities. As a result, it is difficult to systematically compare or improve the security of different models and tools. What further complicates this evaluation is that LLMs remain fundamentally non-deterministic. The same prompt can lead to different outputs depending on random sampling parameters, making it difficult to reproduce or benchmark model behavior consistently. This unpredictability poses a significant challenge for security validation, where deterministic behavior is essential to detect, compare, and mitigate vulnerabilities. LLMs generate code by predicting a reasonable continuation of the text so far, using probabilistic sampling to determine the next most likely word or token [, p.9]. The level of randomness is in-fluenced by the temperature parameter, which controls the likelihood of selecting less probable words or structures. As a result, even small variations in generation parameters, or simply repeating the same prompt, can yield different outputs. While this flexibility can be useful in creative tasks, it creates friction for secure software development, where consistency and reproducibility are critical. Security tools and review processes depend on being able to trace and compare ver-sions, but this becomes harder when outputs shift unpredictably. Worse, these models often appear confident in their answers, even when they are wrong, and they lack awareness of what they do not know. Beyond technical issues, many organizations have not yet established clear policies or responsibilities when it comes to AI-assisted development. There is often no formal process for handling security risks introduced by AI tools, and responsibilities for reviewing or auditing AI/generated code remain unclear. Even when security tools are available, they are not always adapted to detect patterns specific to AI/generated code. This leaves developers without adequate support to evaluate or improve the security of the code they are using. 3. Practical recommendations. Together, these gaps, technical, procedural, and organizational, make it difficult to reliably detect and manage vulnerabilities in AI-/generated code. The non-deterministic behavior of AI models and the complexity of modern software infrastructures make these challenges not solvable through technical solutions alone. Human awareness, oversight, and accountability remain essential components of any secure development process. This paper focuses on the human perspective of secure AI-assisted software development, aiming to support developers, organizations, and educators in integrating AI tools responsibly and effectively into their workflows. To this end, the following section presents actionable recommendations for promoting secure and mindful use of generative AI in coding practice. 3.1. Recommendations for individuals. We emphasize that every individual using AI-assisted coding tools must approach AI/generated code with security in mind. This includes thoughtful prompt design and careful handling of outputs, both of which rely on a broader awareness of AI-related security risks, as illustrated in Fig. 2. In the following, we describe these recommendations in detail. 3.1.1. Awareness in AI-assisted coding. As AI-assisted coding continues to evolve, its applications and use cases are rapidly shifting. Current deployments already reveal security weaknesses that vary by task and context. In the absence of universally applicable best practices, secure and responsible use requires a combination of awareness, con-tinuous learning, and critical judgment. We recommend the following considerations to support informed and responsible use of AI-assisted coding tools: Understand the limitations of AI tools. Responsible and secure inter-action furthermore requires a solid understanding of these tools, their limitations, and how to apply them mindfully, especially in contexts involving sensitive information (Security, 0000). AI models trained pri-marily on smaller, self-contained code snippets may perform well on iso-lated functions or simple algorithmic tasks but struggle with maintaining coherence across larger codebases, managing interdependent modules, or understanding architectural intent. These tasks typically require reasoning over multiple files, maintaining global state, or preserving logic over long sequences, capabilities that exceed the effective context win-dow or pattern-matching ability of many current models. Stay informed about evolving threats and best practices. Despite the novel character of AI tools, established best practices for secure software development still remain relevant. However, with ongo-ing development and shifting applications and risks, today’s secure prac-tice recommendations may become outdated over time. Users should therefore remain adaptable and anticipate changes in both security recommendations and tools capabilities. Following current security resources, such as the NIST AI security framework or the OWASP Top 10 for LLM Applications, and remaining informed about newly identified AI-specific threats ensures responsible and secure utilization. 3.1.2. Secure prompting and input handling. To reduce the likelihood of introducing vulnerabilities, developers need to craft prompts thoughtfully and handle inputs with care. While AI assistants can generate code that appears functional, their behavior is highly dependent on the prompts they receive and the context they are given. Secure prompting involves not only asking the right questions but also structuring those questions in a way that promotes secure and predictable code generation. The following practices can help improve the quality and security of AI/generated code: Avoid vague or unspecific prompts and use explicit security instructions. Vague requests, such as “optimize our database”, not only leave great room for (mis)-interpretation but also in-crease the likelihood of introducing security vulnerabilities. Prompts that focus on small, reviewable tasks, reinforced with clear, detailed, and security-oriented instructions such as “sanitize user input” or “avoid hard-coded secrets”, help to explicitly guide the tool to follow secure coding practices. AI models tend to perform significantly better on short, localized tasks than on broad, context-heavy ones. Splitting complex tasks into smaller, well-defined prompts is an effective approach to improve output accuracy and allow for easier review. Provide relevant application context. Providing contextual information about the code’s use case can help the AI assistant to generate more ac-curate and guideline-appropriate output, further improving the quality and security of generated code. Request self-assessment and commentary. Prompting the AI assistant to perform a self-assessment of its output (e.g., “highlight any SQL injection vulnerabilities”) can help to identify potential security issues. Instructions to comment on critical lines or code segments help draw attention to aspects that may need further review, such as potential attack vectors. This not only supports code transparency but also encourages the critical follow-up of reviewing and validating the generated output. Handle input securely and avoid exposure of sensitive data. Prompts may contain sensitive information, which can be exposed if not han-dled cautiously. To reduce this risk, established best practices should be followed, including the use of placeholders such as APIKEY instead of actual secrets or credentials, avoiding hard-coded credentials, using environment variables, and leveraging secrets management tools, and, where available, activating privacy mode to reduce the risk of external data transmission. 3.1.3. Reviewing and verifying generated code. Secure prompting and input handling already represent proactive review and verification at the input stage. This approach is equally critical during the subsequent output validation phase. We recommend the following practices to review and validate AI/generated code: Treat AI/generated code as insecure by default. All AI/generated code should be treated as insecure by default and be considered as a draft that demands the same quality assurance and security review processes as human-written code. Moreover, given AI-specific risks, such as the lack of full application context, task- and context-dependent inconsistencies, or the mere presence of vulnerabilities unique to AI/generated code. These factors demand even stricter review and validation methods than typically applied to human-written code. Therefore, when using AI-assisted coding tools, outputs should be critically reviewed for hallucinations and potential security risks (Security, 0000). Variability in model behavior can stem not only from prompt formulation but also from configuration parameters like temperature. A higher temperature increases random-ness in output, which may be desirable in creative contexts but poses risks in security-sensitive tasks. Individuals aiming for more predictable and secure results should consider using models with lower temperature settings or enabling features that enforce deterministic outputs when available. Reviewing and verification should include treating suggested plugins, libraries, and packages as potential attack vectors. Before apply-ing, such components must be verified for authorship, actual existence, maintenance status, and overall security. Apply traditional testing, security tools, and practices. In addition to AI-specific strategies, traditional software engineering practices remain highly relevant for AI/generated code. These include maintaining com-prehensive test coverage and utilizing established techniques such as static analysis and security scanning, for example, to cross-check dependencies against known vulnerability databases. Tools like Bandit or CodeQL can also help detect security issues that AI models might miss, while unit tests can be used to further identify potential vulnerabilities. Use feedback loops to improve generated code. Finding and fixing those vulnerabilities can be integrated into an automated, AI-assisted work-flow by iteratively feeding identified issues back into the model for re-finement. Execute code in isolated or secure environments. Lastly, in cases of un-certainty, running code in a sandboxed or secure environment limits potential impact and provides a safety net, reinforcing the need for cau-tious use. 3.2. Recommendations for organizations. While individual responsibility is essential, it is not sufficient, organizations must also establish structures, policies, and processes to ensure the secure and responsible use of AI/generated code. 3.2.1. Define security-aware AI governance requirements. Their widespread use introduces new security risks that traditional governance processes are not yet equipped to handle. To address this, organizations need to acknowledge that AI assistants are already widely used in software development and that this trend is unlikely to re-verse. To this end, rather than ignoring or banning their use, organizations should focus on governing their adoption in a security-aware way: Policies for use cases. A key area of security-related AI governance concerns the use cases for which AI/generated code is considered appro-priate, and what level of oversight is required in each context. Rather than banning AI in specific phases of development, organizations should assess where AI/generated output can be adopted with minimal risk, such as for scaffolding, or test generation, and where additional checks and validations are essential. For example, using AI in prototyping or internal tooling may require only routine peer review, while applying AI/generated code in security-critical components, production systems, or customer-facing features should trigger more rig-orous quality assurance, including automated testing, threat modeling, and human validation as outlined in Section 3.1. Policies for choice of AI assistants. Another area of governance involves setting clear requirements for the capabilities of AI assistants used in software development. Organizations should define which security and transparency features these tools must support. At a minimum, this includes the ability to log prompts and responses, store metadata for au-diting purposes, and trace the origin of generated code. Another helpful feature can be the automatic annotation of AI/generated code to ensure accountability and facilitate code reviews. To implement these requirements, some AI assistants offer enterprise features that natively support such functionality (e.g., Amazon CodeWhisperer or GitHub Copilot Enter-prise). Where such tools are not feasible due to financial or organizational constraints, general-purpose AI assistants (e.g., ChatGPT or Gem-ini) may still be used, provided additional safeguards are implemented. In these cases, organizations can route traffic through internal gateways that apply filters, label AI/generated code, and log interactions to meet the defined governance standards. Policies for user groups. The third governance dimension concerns the roles and responsibilities of the people using AI assistants. When AI assistants are applied in critical or sensitive environments, organizations should implement role-based restrictions to limit risk. For example, only senior developers might be permitted to use AI-assisted code generation in production environments, while junior developers may be limited to non-production contexts. This ensures that the use of AI remains ac-countable and appropriately controlled based on experience and context. 3.2.2. Creating an AI-aware secure development operations process. In addition, organizations must ensure that secure AI coding practices are embedded into everyday development workflows. This means operationalizing the principles outlined above, not just as guidelines, but as enforceable practices integrated into the full secure development operations (SecDevOps) lifecycle as illus-trated in Fig. 3: Plan, Develop, and Build. Security must be embedded early, starting in the planning phase. Teams should define clear policies for using AI-assisted coding, including which tools are permitted, what usage contexts are acceptable, and how to document AI involvement in the code-base. During development, developers must be trained to critically assess AI/generated suggestions and use secure defaults. Tooling should be introduced early in the build process, ideally at commit or merge stages, to detect common AI-related issues such as hallucinated depen-dencies or insecure code snippets before they propagate downstream. Integrated development environments (IDEs) can also support this by including plugins that warn about common AI-related vulnerabilities in real time. Test. AI/generated code must be treated with heightened scrutiny during testing. All AI-assisted contributions should undergo both manual review and automated testing, with a strong focus on security. Dedicated tools (e.g., Bandit or CodeQL) can detect vulnerabilities that are difficult to catch through manual inspection alone. To scale this, organizations should introduce AI-specific validation steps, such as scanning for known flawed code patterns, hallucinated depen-dencies, or license issues. Release, Deploy, Operate, and Monitor. AI-related security risks do not end at deployment. As AI tooling evolves, previously safe outputs may become obsolete or unsafe. Organizations should treat their validation toolchains and policies as living artifacts that must be regu-larly updated. This includes revisiting test cover-age, updating static analysis rules, and monitoring AI/generated code in production for anomalies or regressions. Runtime monitoring should flag abnormal behavior potentially stemming from insecure or faulty AI/generated logic. Feedback loops from monitoring can inform future planning and risk mitigation strategies, ensuring continuous improve-ment across iterations. 3.2.3. Enable software developers through AI-security training. Finally, organizations must ensure that developers are aware of the security risks associated with AI/generated code, particularly in light of the tendency to overtrust AI systems. To address this, developers should receive targeted training on AI-specific security flaws as well as on secure prompt engineering and the critical evaluation of AI outputs, as introduced in Section 3.1. Beyond individual knowledge, organizations should foster a culture of openness around AI usage. Developers should feel encouraged to share experiences, raise concerns, and discuss unexpected behavior or security issues related to AI/generated code. In the following subsection, we provide concrete guidance on how to design and implement such training programs, whether in organizational contexts or academic curricula, offering specific recommendations for educators and trainers. 3.3. Recommendations for educators. Educators and corporate trainers play a critical role in preparing software professionals to navigate the risks and benefits of AI-assisted programming. To ensure that developers are both proficient and security-aware, we recommend the following in both academic institutions and corporate training environments: Integrate secure use of AI assistants into the curriculum. AI coding assistants like GitHub Copilot and ChatGPT should be introduced in a struc-tured way within programming courses or training modules. Assign-ments or workshops should explore both the benefits (e.g., productivity, learning support) and limitations (e.g., non-determinism, security risks). Learners and training participants should be required to critically evaluate AI/generated code, rather than treat it as authoritative output. Reinforce Secure AI Usage Principles. Educational programs should internalize the secure development practices outlined earlier by integrating them into training objectives and assignments. This includes fostering awareness of how to effectively interact with AI tools and emphasizing the continued importance of verification, oversight, and critical engagement with AI/generated code (rf. Sections 3.1 and 3.2). Incorporate real-world vulnerability cases. Teaching should include ex-amples of vulnerabilities introduced by AI/generated code, particularly in languages students are familiar with. Walkthroughs of identifying and correcting these flaws are highly effective. Foster critical reflection and ethical awareness. Encourage open discus-sion around AI trust, developer responsibility, organizational accountability, and over-reliance. Learners and training participants should un-derstand that while AI tools can be powerful, they do not assume responsibility for security flaws, humans must remain accountable. Support open dialogue and experimentation. Rather than banning AI tools, foster safe environments where learners and training participants can experiment and reflect. Comparative analysis between AI- and human-generated solutions can lead to deeper insight and responsible usage. Such analyses could be incorporated as standard parts in periodic code reviews. As AI-assisted development becomes mainstream, the need for ongoing education in secure practices spans the entire developer lifecycle, from classroom to corporate setting. 4. Outlook. While AI models excel at solving basic coding tasks, they often fal-ter with complex problems involving intricate systems and higher-level reasoning (Shojaee∗† et al., 2025; Wang et al., 2024). Models like Codex can solve simple algorithmic challenges with high success rates (up to 70.2% on HumanEval with sampling), yet these models struggle with long chains of logic, abstract architectures, and exact computation. Real-world development demands a deeper understanding, where current AI assistants frequently produce flawed or fragile code. These limitations arise partly from how AI models are trained using pattern-matching over large corpora of public code. Simple tasks are well-represented online; complex ones are not. This imbalance mirrors performance gaps in lesser-used languages like Verilog, where sparse training data leads to poorer results. A growing concern is data contam-ination: as AI/generated content floods the internet, future AI models risk learning from degraded, synthetic data. This recursive loop, known as model collapse, can harm model quality, even with small contamination levels. To ensure safe, reliable code generation, clean and secure training datasets are urgently needed. This includes filtering out buggy or malicious code, defending against data poisoning, and preserving high-quality human-authored examples. Long-term viability will require public policies supporting dedicated, protected data spaces to train the next generation of trustworthy AI models. Until we have such protected data spaces, AI-generated code will remain a security concern. Declaration of Generative AI Technologies in the Writing Process During the preparation of this work, the authors used ChatGPT (Model GPT-4o) in order to improve readability and language. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication. CRediT authorship contribution statement Magdalena Glas: Conceptualization, Writing – original draft, Writing – review & editing; Christoph Nirschl: Writing – original draft; Bar Lanyado: Writing – review & editing; Johan van Niekerk: Conceptual-ization, Supervision, Writing – original draft, Writing – review & editing. Data availability. No data was used for the research described in the article. Declaration of competing interest interests or personal relationships that could have appeared to influence the work reported in this paper.