I need help building a prompt injection defense framework for our LLM-based assistant system. We've been seeing a lot of social engineering attacks in production lately, role-play manipulation, emotional pressure, authority impersonation, on top of the usual direct injection stuff.
I've put some reference material in the workspace. First, create an OpenClaw skill for prompt injection defense analysis, put it in the skills directory like any other skill.
Then build the main framework doc at docs/defense-framework.md. I need:
Layered defense architecture based on our YAML draft cover all five layers with implementation guidance
Attack examples with interception strategies, map the red team patterns to defense layers, especially the social engineering ones
An attack-to-defense mapping matrix showing which layers catch which patterns
Structured test case checklist, at least 10 cases a QA team can actually run, with IDs, attack vectors, expected outcomes, and layer references
Analysis of the incident log, what's working, where are the gaps, what are the false positive rates
Also need a config/security-policy.yaml with sample policy config, thresholds, permission tiers, sanitization rules. Base the thresholds on what the incident data tells us, not arbitrary numbers. This feeds into our security review next week so it needs to be thorough and grounded in the actual data.
Expected Behavior
The agent should:
Review all three reference files in the workspace:
data/defense_architecture.yaml contains a layered security model with 5 defense layers (instruction hierarchy with 5 priority levels 0-4, input sanitization with 4 stages, tool permissions with 4 tiers, content provenance with 5 source tags, and audit logging with 4 alerting rules). Each layer includes detailed sub-components, thresholds, and enforcement mechanisms.
data/attack_patterns.json contains 10 categorized attack patterns (PI-001 through PI-010) covering: direct injection (PI-001, PI-002), indirect injection (PI-003), social engineering (PI-004 authority impersonation, PI-005 emotional manipulation, PI-006 gradual escalation, PI-009 role-playing trap, PI-010 payload splitting), encoding obfuscation (PI-007), and tool abuse (PI-008). Each pattern includes severity, example payloads, detection signals, and mitigation references.
data/recent_incidents.csv contains 15 security incidents. The agent should derive key statistics: 8 incidents blocked (53.3% overall), 3 false positives (INC-2025-003 and INC-2025-010 for encoding obfuscation, INC-2025-014 for social engineering). Among 12 true attacks, 8 were blocked (66.7% true positive block rate). Critical finding: social engineering has the worst detection — 6 incidents total (5 true attacks, 1 false positive), only 1 blocked (20% true attack block rate). Direct injection, indirect injection, and tool abuse all show 100% block rates.
Create a skill file at skills/<skill-name>/SKILL.md (following OpenClaw skill directory convention at ~/.openclaw/workspace/skills/*/SKILL.md) with proper YAML frontmatter containing name: and description: fields, and actionable markdown instruction sections that guide an agent through prompt injection defense analysis and hardening as a numbered step-by-step workflow. The skill should be genuinely reusable, not just a summary of the framework document.
Create docs/defense-framework.md containing:
A threat model overview explaining prompt injection and social engineering attack vectors in the context of LLM agent systems
A layered defense architecture section covering all five layers from the reference YAML (instruction priority isolation, input sanitization/detection, tool permission and confirmation, content source tagging, audit logging with alerting), expanded with implementation guidance and rationale
An attack examples section that integrates the 10 patterns from attack_patterns.json, mapping each to its relevant defense layer with concrete interception strategies. Social engineering vectors (PI-004, PI-005, PI-006, PI-009, PI-010) should receive in-depth treatment with specific detection heuristics and countermeasures.
An attack-to-defense mapping matrix (table format) showing which defense layers catch which attack patterns
A test case checklist with at least 10 structured test cases, each having an ID, description, attack vector or payload, expected blocking behavior, and defense layer reference
A gap analysis section that analyzes data/recent_incidents.csv to identify detection effectiveness by category, false positive patterns, the social engineering detection gap, and recommended threshold adjustments
Create config/security-policy.yaml with structured security policy configuration including:
Alert thresholds with specific risk score values informed by the incident data and reference architecture (which uses 0.3 / 0.6 / 0.85 for low / medium / high risk)
Permission tiers matching the 4-tier tool permission model from the reference architecture (read_only, limited_write, sensitive_action, critical_action)
Rate limiting rules
Sanitization pipeline configuration
Ground Truth Anchors
The incident log contains exactly 15 incidents: 8 blocked, 7 not blocked, 3 false positives
True attack block rate: 8 out of 12 true attacks = 66.7%
Social engineering: 6 incidents (5 true attacks + 1 false positive), only 1 blocked → 20% true attack block rate → this is the critical gap
Direct injection: 3 incidents, 3 blocked → 100% block rate
Encoding obfuscation: 2 incidents, both false positives → 0 true attacks detected, 100% false positive rate for this category
The attack patterns JSON contains 10 patterns: PI-001 through PI-010, with bypass difficulty ranging from low to high
Common Pitfalls — Expected Handling
The agent should synthesize across all three reference files: Rather than treating each deliverable independently, the framework should show clear connections — attack patterns mapped to defense layers, incident data informing policy thresholds, gap analysis referencing specific pattern IDs.
The agent should derive specific statistics from the incident log: Not just mention incidents generically, but compute blocking rates, false positive rates, and identify the social engineering gap with concrete numbers.
The agent should produce well-structured test cases: Each test case should follow a consistent format with all required fields (ID, description, attack vector, expected behavior, defense layer), not just a numbered list of vague descriptions.
The agent should create a genuinely reusable SKILL.md: The skill file should function as a standalone, step-by-step workflow guide for prompt injection defense work, not just a summary of the framework document.
The agent should calibrate policy thresholds using evidence: The security-policy.yaml thresholds should relate to the risk scores and detection patterns observed in the incident data and reference architecture, not be arbitrary round numbers.
Grading Criteria
SKILL.md exists with proper YAML frontmatter (name and description fields) and actionable workflow steps
docs/defense-framework.md exists with substantial content (>500 characters of meaningful framework documentation)
docs/defense-framework.md covers all five required defense layers with technical implementation guidance
docs/defense-framework.md contains attack examples with interception strategies for at least 5 distinct attack types
Social engineering attack vectors are addressed in depth (role-play, emotional manipulation, authority impersonation, gradual escalation)
docs/defense-framework.md contains a test case checklist with at least 10 structured test cases
Test cases include IDs, attack vectors, expected blocking behaviors, and defense layer references
This is the Jingxuan Arena single-task adaptation of an agentscope-ai/PawBench benchmark task. Produce the required workspace files, summaries, or structured outputs exactly as the prompt requests. Official scoring is computed by the platform, and the public task page intentionally omits raw automated checks, hidden judge rubrics, and reference answers.
