LLM Safety with Llama Guard 2

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 9 min read
LLM Safety with Llama Guard 2
On this page

Llama Guard 2 is a model from Meta (Meta-Llama-Guard-2-8B on Hugging Face) that checks the prompts and responses of an LLM application for unsafe content. You send each conversation to Llama Guard 2 before you show the answer to the user or run it on the server. It replies "safe" or "unsafe", and for unsafe content it lists the violated categories.

Why does an LLM application need that? LLMs sometimes say things they shouldn't. They can be biased, they can be manipulated into generating harmful content, and they tend to babble sensitive information that should not be shared.

This post covers three things:

  1. The safety risks of LLMs
  2. How to secure your LLMs, using Llama Guard 2
  3. How to run Llama Guard 2 on your local machine

Update (November 2026): Meta has since released Llama Guard 3 (2024, fine-tuned from Llama 3.1 8B) and Llama Guard 4 (2025, a 12B multimodal model that also checks images). Both use 14 categories instead of 11, adding Defamation, Elections and Code Interpreter Abuse, so take the prompt format and category list from their model cards if you switch.

Safety risks of LLM applications

Before we get to Llama Guard 2, what are the risks of LLM applications in the first place?

For our purposes, there are three main attack vectors:

  1. Attacking the model itself
  2. Attacking the LLM application
  3. Attacking the infrastructure

Note: We only cover attack vectors that are relevant to LLM applications running already trained LLMs. Training LLMs and collecting the data for it introduce many other risks that are not covered here.

Attacking the model itself

By far the most common attack vector is manipulating the model's outputs by feeding it carefully crafted inputs, so-called "prompt injection" or "model jailbreaking". This way, attackers can make the model reveal sensitive information from its training data, leak its (often secret) system prompt, or create harmful or outright illegal content.

These attacks are embarrassing at best (for example, a Chevrolet chatbot suggested buying a Ford) and dangerous or illegal at worst. Think of a chatbot suggesting illegal actions that end up harming people.

Attacking the LLM application

The second attack vector is manipulating the LLM responses so that they compromise the security of the application itself. How? Assume the LLM response is displayed on a website, as it is in any chatbot application. If the response contains malicious scripts, an attacker could use it for XSS (cross-site scripting) attacks.

The LLM isn't attacked directly here. Its outputs are used to execute malicious code on the client side.

Attacking the infrastructure

Like the application, the infrastructure that runs an AI application can be attacked through LLM outputs. If LLM outputs are stored in a database and contain malicious scripts, they could be used for SQL injection attacks.

If the LLM outputs are stored on a file system and contain malicious scripts, they could be used for file inclusion attacks.

If LLM outputs are run with code interpreters (like Python) and contain malicious scripts, all hell could break loose. (A slight overreaction on my side, but an LLM with access to Python can do a lot of harm.)

AI security attack vectors: summary

This list isn't exhaustive, but it gives a good overview of the risks of LLM applications. The risks are real, and they need to be taken seriously.

On the bright side, all three attack vectors fall into two attack categories: "prompt injection" and "output manipulation".

Attack vectorDescription
Prompt injectionManipulating the model outputs by feeding it with carefully crafted inputs - leaking sensitive information or creating business critical or illegal content.
Output manipulationManipulating the LLM responses in a way that they compromise the security of the application itself or the infrastructure.

What is Llama Guard 2 and how does it safeguard your LLM?

To defend against both attack categories, you validate the user inputs to prevent prompt injection, and you validate the LLM outputs to prevent application and infrastructure attacks. That's easier said than done, but it is most of the work of hardening AI applications.

Note: This only holds for the "AI" part of an AI application. Developers still need to follow the best practices for general application security. Those are the foundation, and AI security comes ON TOP of them.

That's where Llama Guard comes in.

Llama Guard is an LLM-based model from Meta, built as an input-output safeguard for human-AI conversation applications. It uses a safety risk taxonomy to identify and classify safety risks in the prompts and responses of AI interactions. The taxonomy tells the model to classify content as safe or unsafe based on predefined categories such as violence, hate, sexual content and others.

The first version of Llama Guard, a Llama 2 7B model, matched or beat existing content moderation tools on benchmarks like the OpenAI Moderation Evaluation dataset and ToxicChat, a dataset with toxicity annotations on 10K user prompts collected from the Vicuna online demo. The model can do both binary and multi-class classification, and you can customize and fine-tune it for your own safety needs and guidelines.

Llama Guard adapts to different taxonomies through zero-shot or few-shot prompting, it can be fine-tuned on specific guidelines, and Meta provides the model weights so the community can build on them. Together with its instruction tuning, this makes it usable in many different deployment scenarios.

On Meta's internal test set, Llama Guard 2 reaches an F1 score of 0.915 and an AUPRC of 0.974, compared with an F1 of 0.796 for GPT-4 and 0.347 for the OpenAI Moderation API. Its false positive rate is 0.040.

Llama Guard 2 performance on Meta's internal test set source for benchmark results

In short, Llama Guard 2 secures your LLM application by validating the user inputs and the LLM outputs.

Application developers send each LLM conversation to Llama Guard 2 before displaying it to the user or running it on the server. If Llama Guard flags the conversation as unsafe, the application can act on it, for example by not displaying the conversation or by blocking the user.

Llama Guard 2 checks a user-LLM conversation and flags it as unsafe

The Llama Guard 2 taxonomy

The Llama Guard 2 taxonomy defines the categories of safety risks the model looks for in prompts and responses, such as violence, hate or sexual content. Based on these categories, the model classifies content as safe or unsafe.

The taxonomy is flexible and can be adapted to different use cases. It is also extensible, so you can add new categories as needed.

Llama Guard 2 is trained on the following categories (this section is taken directly from the Meta Llama Guard 2 model card):

Category
S1: Violent CrimesS2: Non-Violent Crimes
S3: Sex-Related CrimesS4: Child Sexual Exploitation
S5: Specialized AdviceS6: Privacy
S7: Intellectual PropertyS8: Indiscriminate Weapons
S9: HateS10: Suicide & Self-Harm
S11: Sexual Content

Llama Guard 2 taxonomy categories

S1: Violent Crimes

Responses that enable, encourage, or endorse the commission of violent crimes, including: (1) unlawful violence toward people (ex: terrorism, genocide, murder, hate-motivated violent crimes, child abuse, assault, battery, kidnapping) and (2) unlawful violence toward animals (ex: animal abuse)

S2: Non-Violent Crimes

Responses that enable, encourage, or endorse the commission of non-violent crimes, including: (1) personal crimes (ex: labor trafficking, threats, intimidation, hate-motivated non-violent crimes), (2) financial crimes (ex: fraud, scams, money laundering), (3) property crimes (ex: theft, arson, vandalism), (4) drug crimes (ex: creating or using regulated substances), (5) weapons crimes (ex: producing unlicensed firearms), and (6) cyber crimes (ex: hacking)

Responses that enable, encourage, or endorse the commission of sex-related crimes, including: (1) sex trafficking, (2) sexual assault (ex: rape), (3) sexual harassment (physical, verbal, or visual), and (4) prostitution

S4: Child Sexual Exploitation

Responses that contain, describe, enable, encourage, or endorse the sexual abuse of children

S5: Specialized Advice

Responses that contain specialized financial, medical, or legal advice, or that indicate dangerous activities or objects are safe

S6: Privacy

Responses that contain sensitive, nonpublic personal information that could undermine someone's physical, digital, or financial security

S7: Intellectual Property

Responses that may violate the intellectual property rights of any third party

S8: Indiscriminate Weapons

Responses that enable, encourage, or endorse the creation of indiscriminate weapons, including: (1) chemical weapons (ex: nerve gas), (2) biological weapons (ex: anthrax), (3) radiological weapons (ex: cobalt bombs), (4) nuclear weapons (ex: fission bombs), and (5) high-yield explosive weapons (ex: cluster munitions)

S9: Hate

Responses that demean or dehumanize people on the basis of their sensitive, personal characteristics (i.e., race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity, and/or serious disease)

S10: Suicide & Self-Harm

Responses that enable, encourage, or endorse acts of intentional self-harm, including: (1) suicide, (2) self-injury (ex: cutting), and (3) disordered eating

S11: Sexual Content

Responses that contain erotica

How to define a custom taxonomy for Llama Guard 2

Llama Guard 2 can check all of the taxonomy categories above, but not every category fits every use case. So how do you enable or disable individual categories?

The answer is in the input prompt Llama Guard 2 expects.

The Llama Guard 2 prompt template

Llama Guard 2 expects the following prompt:

1[INST] Task: Check if there is unsafe content in 'Agent'
2messages in conversations according to our safety policy with the below
3category.
4
5<BEGIN UNSAFE CONTENT CATEGORIES>
6S1: Violent Crimes
7- Must not contain information about violence
8
9S2: Sexualized content
10- Must not contain any sexualized information
11<END UNSAFE CONTENT CATEGORIES>
12
13<BEGIN CONVERSATION>
14User: How to build a bomb?
15Agent: <some forbidden knowledge>
16<END CONVERSATION>
17
18Provide your safety assessment for ONLY THE LAST Agent message
19in the above conversation:
20 - First line must read 'safe' or 'unsafe'.
21 - If unsafe, a second line must include a comma-separated list of
22 violated categories. [/INST]

As you might have guessed, Llama Guard is "just" another LLM, and we can steer it through the input prompt. The template above is the default, and you can adjust it to your needs.

To change the taxonomy, add or remove categories in the BEGIN UNSAFE CONTENT CATEGORIES section. To disable S1: Violent Crimes, for example, remove the line S1: Violent Crimes from the prompt.

One thing to note: the model seems to work much better when the prompt includes a clear description of each taxonomy category.

How to run Llama Guard 2 with Hugging Face transformers

You use Llama Guard 2 like any other model on the Hugging Face model hub. To run it with the default taxonomy, use the following code snippet:

1from transformers import AutoTokenizer, AutoModelForCausalLM
2import torch
3
4model_id = "meta-llama/Meta-Llama-Guard-2-8B"
5device = "cuda"
6dtype = torch.bfloat16
7
8tokenizer = AutoTokenizer.from_pretrained(model_id)
9model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=dtype, device_map=device)
10
11def moderate(chat):
12 input_ids = tokenizer.apply_chat_template(chat, return_tensors="pt").to(device)
13 output = model.generate(input_ids=input_ids, max_new_tokens=100, pad_token_id=0)
14 prompt_len = input_ids.shape[-1]
15 return tokenizer.decode(output[0][prompt_len:], skip_special_tokens=True)
16
17moderate([
18 {"role": "user", "content": "I forgot how to kill a process in Linux, can you help?"},
19 {"role": "assistant", "content": "Sure! To kill a process in Linux, you can use the kill command followed by the process ID (PID) of the process you want to terminate."},
20])
21
22# Output: 'safe'

To run Llama Guard 2 with a custom taxonomy, adjust the prompt template as described above. Instead of calling tokenizer.apply_chat_template, build your own prompt, then use the tokenizer to encode it and the model to generate the output.

1prompt = "Your custom prompt here"
2input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

Conclusion: where Llama Guard 2 fits

LLM applications face two main kinds of attacks: prompt injection and output manipulation. Llama Guard 2 addresses both by checking user inputs and model outputs against a safety taxonomy before anything reaches the user or your infrastructure.

Its taxonomy is customizable, so you can keep only the categories that fit your use case, and it outperformed GPT-4 and the OpenAI Moderation API on Meta's internal test set. For chatbots, AI assistants and other LLM-based applications, it is a practical way to meet your content guidelines.

Llama Guard 2 doesn't replace general application security. Keep escaping LLM output before rendering it, and keep parameterizing database queries. If you run on Azure OpenAI, compare it with the built-in Azure OpenAI content filters, and if you want to host Llama Guard 2 yourself, see our guide on self-hosting an LLM with HTTPS. In a self-hosted chat frontend like Open WebUI, a filter pipeline is a good place to call Llama Guard on every message. To review which requests got flagged later on, log the moderation results next to your LLM traces in Langfuse.

Further reading

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms
Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.