LLM Safety with Llama Guard 2

On this page
Llama Guard 2 is a model from Meta (Meta-Llama-Guard-2-8B on Hugging Face)
that checks the prompts and responses of an LLM application for unsafe
content. You send each conversation to Llama Guard 2 before you show the
answer to the user or run it on the server. It replies "safe" or "unsafe",
and for unsafe content it lists the violated categories.
Why does an LLM application need that? LLMs sometimes say things they shouldn't. They can be biased, they can be manipulated into generating harmful content, and they tend to babble sensitive information that should not be shared.
This post covers three things:
- The safety risks of LLMs
- How to secure your LLMs, using Llama Guard 2
- How to run Llama Guard 2 on your local machine
Update (November 2026): Meta has since released Llama Guard 3 (2024, fine-tuned from Llama 3.1 8B) and Llama Guard 4 (2025, a 12B multimodal model that also checks images). Both use 14 categories instead of 11, adding Defamation, Elections and Code Interpreter Abuse, so take the prompt format and category list from their model cards if you switch.
Safety risks of LLM applications
Before we get to Llama Guard 2, what are the risks of LLM applications in the first place?
For our purposes, there are three main attack vectors:
- Attacking the model itself
- Attacking the LLM application
- Attacking the infrastructure
Note: We only cover attack vectors that are relevant to LLM applications running already trained LLMs. Training LLMs and collecting the data for it introduce many other risks that are not covered here.
Attacking the model itself
By far the most common attack vector is manipulating the model's outputs by feeding it carefully crafted inputs, so-called "prompt injection" or "model jailbreaking". This way, attackers can make the model reveal sensitive information from its training data, leak its (often secret) system prompt, or create harmful or outright illegal content.
These attacks are embarrassing at best (for example, a Chevrolet chatbot suggested buying a Ford) and dangerous or illegal at worst. Think of a chatbot suggesting illegal actions that end up harming people.
Attacking the LLM application
The second attack vector is manipulating the LLM responses so that they compromise the security of the application itself. How? Assume the LLM response is displayed on a website, as it is in any chatbot application. If the response contains malicious scripts, an attacker could use it for XSS (cross-site scripting) attacks.
The LLM isn't attacked directly here. Its outputs are used to execute malicious code on the client side.
Attacking the infrastructure
Like the application, the infrastructure that runs an AI application can be attacked through LLM outputs. If LLM outputs are stored in a database and contain malicious scripts, they could be used for SQL injection attacks.
If the LLM outputs are stored on a file system and contain malicious scripts, they could be used for file inclusion attacks.
If LLM outputs are run with code interpreters (like Python) and contain malicious scripts, all hell could break loose. (A slight overreaction on my side, but an LLM with access to Python can do a lot of harm.)
AI security attack vectors: summary
This list isn't exhaustive, but it gives a good overview of the risks of LLM applications. The risks are real, and they need to be taken seriously.
On the bright side, all three attack vectors fall into two attack categories: "prompt injection" and "output manipulation".
| Attack vector | Description |
|---|---|
| Prompt injection | Manipulating the model outputs by feeding it with carefully crafted inputs - leaking sensitive information or creating business critical or illegal content. |
| Output manipulation | Manipulating the LLM responses in a way that they compromise the security of the application itself or the infrastructure. |
What is Llama Guard 2 and how does it safeguard your LLM?
To defend against both attack categories, you validate the user inputs to prevent prompt injection, and you validate the LLM outputs to prevent application and infrastructure attacks. That's easier said than done, but it is most of the work of hardening AI applications.
Note: This only holds for the "AI" part of an AI application. Developers still need to follow the best practices for general application security. Those are the foundation, and AI security comes ON TOP of them.
That's where Llama Guard comes in.
Llama Guard is an LLM-based model from Meta, built as an input-output safeguard for human-AI conversation applications. It uses a safety risk taxonomy to identify and classify safety risks in the prompts and responses of AI interactions. The taxonomy tells the model to classify content as safe or unsafe based on predefined categories such as violence, hate, sexual content and others.
The first version of Llama Guard, a Llama 2 7B model, matched or beat existing content moderation tools on benchmarks like the OpenAI Moderation Evaluation dataset and ToxicChat, a dataset with toxicity annotations on 10K user prompts collected from the Vicuna online demo. The model can do both binary and multi-class classification, and you can customize and fine-tune it for your own safety needs and guidelines.
Llama Guard adapts to different taxonomies through zero-shot or few-shot prompting, it can be fine-tuned on specific guidelines, and Meta provides the model weights so the community can build on them. Together with its instruction tuning, this makes it usable in many different deployment scenarios.
On Meta's internal test set, Llama Guard 2 reaches an F1 score of 0.915 and an AUPRC of 0.974, compared with an F1 of 0.796 for GPT-4 and 0.347 for the OpenAI Moderation API. Its false positive rate is 0.040.
Llama Guard 2 performance on Meta's internal test set source for benchmark results
In short, Llama Guard 2 secures your LLM application by validating the user inputs and the LLM outputs.
Application developers send each LLM conversation to Llama Guard 2 before displaying it to the user or running it on the server. If Llama Guard flags the conversation as unsafe, the application can act on it, for example by not displaying the conversation or by blocking the user.
Llama Guard 2 checks a user-LLM conversation and flags it as unsafe
The Llama Guard 2 taxonomy
The Llama Guard 2 taxonomy defines the categories of safety risks the model looks for in prompts and responses, such as violence, hate or sexual content. Based on these categories, the model classifies content as safe or unsafe.
The taxonomy is flexible and can be adapted to different use cases. It is also extensible, so you can add new categories as needed.
Llama Guard 2 is trained on the following categories (this section is taken directly from the Meta Llama Guard 2 model card):
| Category | |
|---|---|
| S1: Violent Crimes | S2: Non-Violent Crimes |
| S3: Sex-Related Crimes | S4: Child Sexual Exploitation |
| S5: Specialized Advice | S6: Privacy |
| S7: Intellectual Property | S8: Indiscriminate Weapons |
| S9: Hate | S10: Suicide & Self-Harm |
| S11: Sexual Content | |
Llama Guard 2 taxonomy categories
S1: Violent Crimes
Responses that enable, encourage, or endorse the commission of violent crimes, including: (1) unlawful violence toward people (ex: terrorism, genocide, murder, hate-motivated violent crimes, child abuse, assault, battery, kidnapping) and (2) unlawful violence toward animals (ex: animal abuse)
S2: Non-Violent Crimes
Responses that enable, encourage, or endorse the commission of non-violent crimes, including: (1) personal crimes (ex: labor trafficking, threats, intimidation, hate-motivated non-violent crimes), (2) financial crimes (ex: fraud, scams, money laundering), (3) property crimes (ex: theft, arson, vandalism), (4) drug crimes (ex: creating or using regulated substances), (5) weapons crimes (ex: producing unlicensed firearms), and (6) cyber crimes (ex: hacking)
S3: Sex-Related Crimes
Responses that enable, encourage, or endorse the commission of sex-related crimes, including: (1) sex trafficking, (2) sexual assault (ex: rape), (3) sexual harassment (physical, verbal, or visual), and (4) prostitution
S4: Child Sexual Exploitation
Responses that contain, describe, enable, encourage, or endorse the sexual abuse of children
S5: Specialized Advice
Responses that contain specialized financial, medical, or legal advice, or that indicate dangerous activities or objects are safe
S6: Privacy
Responses that contain sensitive, nonpublic personal information that could undermine someone's physical, digital, or financial security
S7: Intellectual Property
Responses that may violate the intellectual property rights of any third party
S8: Indiscriminate Weapons
Responses that enable, encourage, or endorse the creation of indiscriminate weapons, including: (1) chemical weapons (ex: nerve gas), (2) biological weapons (ex: anthrax), (3) radiological weapons (ex: cobalt bombs), (4) nuclear weapons (ex: fission bombs), and (5) high-yield explosive weapons (ex: cluster munitions)
S9: Hate
Responses that demean or dehumanize people on the basis of their sensitive, personal characteristics (i.e., race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity, and/or serious disease)
S10: Suicide & Self-Harm
Responses that enable, encourage, or endorse acts of intentional self-harm, including: (1) suicide, (2) self-injury (ex: cutting), and (3) disordered eating
S11: Sexual Content
Responses that contain erotica
How to define a custom taxonomy for Llama Guard 2
Llama Guard 2 can check all of the taxonomy categories above, but not every category fits every use case. So how do you enable or disable individual categories?
The answer is in the input prompt Llama Guard 2 expects.
The Llama Guard 2 prompt template
Llama Guard 2 expects the following prompt:
As you might have guessed, Llama Guard is "just" another LLM, and we can steer it through the input prompt. The template above is the default, and you can adjust it to your needs.
To change the taxonomy, add or remove categories in the BEGIN UNSAFE CONTENT CATEGORIES section. To disable S1: Violent Crimes, for example,
remove the line S1: Violent Crimes from the prompt.
One thing to note: the model seems to work much better when the prompt includes a clear description of each taxonomy category.
How to run Llama Guard 2 with Hugging Face transformers
You use Llama Guard 2 like any other model on the Hugging Face model hub. To run it with the default taxonomy, use the following code snippet:
To run Llama Guard 2 with a custom taxonomy, adjust the prompt template as
described above. Instead of calling tokenizer.apply_chat_template, build
your own prompt, then use the tokenizer to encode it and the model to
generate the output.
Conclusion: where Llama Guard 2 fits
LLM applications face two main kinds of attacks: prompt injection and output manipulation. Llama Guard 2 addresses both by checking user inputs and model outputs against a safety taxonomy before anything reaches the user or your infrastructure.
Its taxonomy is customizable, so you can keep only the categories that fit your use case, and it outperformed GPT-4 and the OpenAI Moderation API on Meta's internal test set. For chatbots, AI assistants and other LLM-based applications, it is a practical way to meet your content guidelines.
Llama Guard 2 doesn't replace general application security. Keep escaping LLM output before rendering it, and keep parameterizing database queries. If you run on Azure OpenAI, compare it with the built-in Azure OpenAI content filters, and if you want to host Llama Guard 2 yourself, see our guide on self-hosting an LLM with HTTPS. In a self-hosted chat frontend like Open WebUI, a filter pipeline is a good place to call Llama Guard on every message. To review which requests got flagged later on, log the moderation results next to your LLM traces in Langfuse.
Further reading
- How to integrate knowledge with LLMs?
- How to test your RAG pipeline?
- How to host your own LLM - including HTTPS?
- How to trace and evaluate LLM applications with Opik
Interested in building high-quality AI agent systems?
We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:
- Understanding the 14 systematic failure modes in multi-agent systems
- Evidence-based best practices for agent design
- Structured communication protocols and verification mechanisms



