Auditing the security of LLMs

Large Language Models (LLMs) are becoming a standard component of information systems, but very few organizations have yet integrated these technologies into their technical auditing and penetration testing practices. This article proposes a forward-looking approach. cyber audit, pentesting And offensive security to concretely assess the risks associated with LLMs, without limiting oneself to a simple theoretical review of the OWASP Top 10.

1. Why are LLMs revolutionizing technical auditing?

LLMs are not like a "simple" web service: they combine a statistical model, opaque training data, complex prompts, business connectors, and sometimes agents capable of executing actions. In a cybersecurity audit context, this means that the attack surface covers the front end, the APIs, the orchestration engine, the model, and the MLOps pipelines that feed it.

For a team of security or pentesting, The impacts are tangible:

• Penetration testing can no longer stop at the interface: it is also necessary to test the prompt logic, the knowledge sources, the tools and the automated workflows.
• Traditional risks (XSS, injections, data exfiltration) are combined with new scenarios specific to generative AI: poisoning, model theft (Or model extraction) And adversarial attacks.

2. New risks and reference frameworks

The arrival of LLM necessitates a reinterpretation of existing security standards. Frameworks such as the’OWASP Top 10 for LLM Applications, THE NIST AI RMF or the recommendations of the’ANSSI constitute a useful working basis, but insufficient to cover all operational risks related to orchestration, prompts and MLOps pipelines.
An effective LLM audit must therefore combine compliance analysis, grey box tests And simulations of adversary attacks to assess the overall resilience of the system.

3. Map the attack surfaces before the penetration test

Before launching a LLM pentesting, The first step is a precise mapping of the application and its ecosystem. Without this step, the audit would be limited to surface tests (e.g., simple chatbot) and could miss the real issues (data, automation, integrations).

Key aspects of the mapping:

• Data feeds and prompts What types of data are transmitted through the prompts (customers, HR, internal IS, logs)? Is there a RAG mechanism, and how are the sources managed (access rights, updates, versioning)?
• Connectors and tools What systems can the agent or chatbot query or modify (ERP, CRM, ticketing, technical repositories)? What automation gateways (orchestrators, webhooks, scripts) are triggered by the LLM output?
• MLOps Pipeline How is the model trained, tuned, deployed, and versioned? Who can modify the datasets, system prompts, or policies?

This map serves as the basis for the perimeter of’technical audit and of’cyber audit : it allows you to define the components to target, the environments to test and the datasets to anonymize.

4. LLM pentesting methodology: three complementary blocks

4.1. Model-oriented and prompt tests

The objective is to verify the system's resilience to malicious or unforeseen interactions on the LLM side.

• Attacks on the content of the prompt : attempts to circumvent security instructions (prompt injection, jailbreak).
• Exfiltration tests : push the model to reveal internal information (training data, secrets, system prompts, internal rules).
• Response integrity tests : analyze the model's ability to reject malicious solicitations (attack code requests, illicit content).
• Hallucination assessment in critical use cases (legal, compliance, finance).

The results document the robustness of the system against textual attacks and help to define when human validation remains essential.

4.2. Integration-oriented testing and toolchain

This layer of the pentesting aims to understand how the application consumes the model outputs and secures the exposed tools.

• LLM output consumption : verification of resistance to HTML/JS injections on the client or server side; analysis of workflows that execute commands or call APIs based on model responses.
• Plugins, tools and connectors : tests for forcing unauthorized calls, verification of access controls (authentication, authorization, parameter validation).

This phase brings the LLM penetration test from an advanced application pentest, enriched by the probabilistic logic inherent in the model.

4.3. MLOps-oriented testing and governance

The technical audit must finally cover the complete lifecycle of the model and data.

• Data Poisoning and Integrity : analysis of training sources, quality control and access management. Verification of protections against malicious data injection into pipelines or external document databases.
• Protection of the model and artifacts : control of access to model weights, verification of access logs, anti-extraction devices (rate limiting, query monitoring).

This layer extends the practices DevSecOps to the universe MLOps, by integrating security from the training and deployment operations.

5. Transform audit results into a remediation plan

An LLM audit is only valuable if it leads to a clear and prioritized action plan:

• Securing data flows and data : reduction of sensitive data sent to the model, compartmentalization by perimeter, systematic anonymization.
• Strengthening of integrations : filters and validations on inputs/outputs, strong authentication of plugins, fine control of rights and limitation of actions.
• Governance and oversight : integration of LLM risks into the overall mapping, regular review of prompts, system, models and data.

The goal: to move from a one-off audit to a continuous improvement loop, in line with existing practices (SOC, vulnerability management, architecture magazine).

6. Towards a recurring LLM audit program

With the widespread adoption of LLM in business processes, the question is no longer «"Should we conduct an audit?"» but «"At what pace and over what area?"»
A mature program combines:

• Regular audits and penetration tests on the new LLM implementations.
• Integrated CI/CD and MLOps Controls to monitor drift, data quality and new attack vectors.
• Upskilling of the SOC, GRC and Dev teams according to executives ANSSI And NIST.

Conclusion: Secure the use of AI within your organization

LLM is no longer an experimental topic: it is now at the heart of your business processes and customer interactions.
A cyber audit or a penetration test Specialized generative AI has become necessary to avoid compromises and non-compliance.

Intrinsic, expert in offensive security For 30 years, we have been helping you transform these risks into levers of resilience:

• Technical audit and LLM penetration testing to map and prioritize your vulnerabilities.
• Red Team and Purple Team to assess your maturity against advanced AI attacks.
• Strategic support: MLOps governance, SOC integration, ANSSI compliance and EU AI Act.