CVE/Security Triage

Prompt

You are a security engineer triaging the results of a dependency vulnerability scan for your company's production web application. The scan results are in vulnerability_scan.json and the deployment context is in deployment_context.md.

Review all 10 CVEs in the scan report and produce two deliverables:

1. triage_report.md — A triage report that classifies each CVE by operational priority (P0 through P3, using the change management policy in deployment_context.md), ordered from most critical to least. For each CVE, include:

   • CVE ID, package name, and CVSS score
   • Your assigned priority (P0/P1/P2/P3)
   • A 1-2 sentence justification for the priority assignment considering both the CVSS severity and the deployment context (e.g., whether the vulnerable component is in the runtime vs. build tooling, whether exploit mitigations exist)
   • Whether the CVE affects PCI DSS scope

2. remediation_plan.md — A remediation plan that groups the CVEs into actionable work items, considering:

   • Which patches can be batched together (e.g., same upgrade path or same priority tier)
   • The deploy windows available (April 14 and April 17)
   • Any testing requirements before deploying (especially for breaking changes like major version bumps)
   • A specific timeline mapping patches to deploy windows
   • Risk notes for any CVE where the upgrade involves a major version change

Include an executive summary at the top of the remediation plan suitable for sharing with leadership.

Expected Behavior

The agent should:

1. Read and parse vulnerability_scan.json to understand all 10 CVEs
2. Read deployment_context.md to understand the deployment context, change management policy, and security posture
3. Apply contextual reasoning to map CVSS severity to operational priority:
   • CVE-2026-29112 (express RCE, CVSS 9.8, public PoC) → P0
   • CVE-2026-31845 (JWT bypass, CVSS 9.1, actively exploited) → P0
   • CVE-2026-27519 (SQL injection, CVSS 8.1, affects PCI scope) → P0 or P1
   • CVE-2026-28003 (lodash prototype pollution, CVSS 7.5) → P1
   • CVE-2026-30291 (axios SSRF, CVSS 7.4) → P1
   • CVE-2026-25877 (helmet CSP bypass, CVSS 5.4, runtime) → P2
   • CVE-2026-33010 (minimatch ReDoS, CVSS 5.3, build-only) → P2 or P3
   • CVE-2026-34201 (semver ReDoS, CVSS 5.3, build-only) → P2 or P3
   • CVE-2026-36500 (debug info leak, CVSS 3.1, transitive) → P3
   • CVE-2026-37188 (qs DoS, CVSS 3.7, transitive, conditional) → P3
4. Identify that the JWT vulnerability is actively exploited and warrants immediate action
5. Note PCI DSS implications for payment-related components (the pg SQL injection and JWT bypass both affect authentication/payment flows)
6. Create a remediation plan that maps patches to the April 14 and April 17 deploy windows
7. Flag major version upgrades (jsonwebtoken 8→9, helmet 4→7, debug 2→4, qs 6.5→6.12) as requiring additional testing

Acceptable variations:

• The pg SQL injection (CVE-2026-27519) may reasonably be classified as P0 or P1 depending on how the agent weighs the authenticated-access prerequisite vs. the PCI scope
• Build-only vulnerabilities (minimatch, semver) may be P2 or P3
• The agent may propose different batching strategies as long as they respect priority ordering

Grading Criteria

• triage_report.md is created
• remediation_plan.md is created
• All 10 CVEs appear in the triage report
• Each CVE has a priority assigned (P0-P3)
• Express RCE (CVE-2026-29112) is classified as P0
• JWT bypass (CVE-2026-31845) is classified as P0
• Build-only CVEs (minimatch, semver) are classified lower than runtime CVEs
• PCI DSS scope is mentioned for relevant CVEs
• Remediation plan includes a timeline with deploy windows
• Major version upgrades are flagged as requiring extra testing
• Executive summary exists in remediation plan

Automated Checks

def grade(transcript: list, workspace_path: str) -> dict:
    """
    Grade the CVE/Security Triage task.

    Checks for structural completeness of both deliverables and
    correctness of key priority assignments.
    """
    from pathlib import Path
    import re

    scores = {}
    workspace = Path(workspace_path)

    triage_file = workspace / "triage_report.md"
    remediation_file = workspace / "remediation_plan.md"

    # --- Check 1: triage_report.md created ---
    if not triage_file.exists():
        scores["triage_report_created"] = 0.0
    else:
        scores["triage_report_created"] = 1.0

    # --- Check 2: remediation_plan.md created ---
    if not remediation_file.exists():
        scores["remediation_plan_created"] = 0.0
    else:
        scores["remediation_plan_created"] = 1.0

    # Early return if neither file exists
    if not triage_file.exists() and not remediation_file.exists():
        return {
            "triage_report_created": 0.0,
            "remediation_plan_created": 0.0,
            "all_cves_covered": 0.0,
            "priorities_assigned": 0.0,
            "express_rce_is_p0": 0.0,
            "jwt_bypass_is_p0": 0.0,
            "build_only_lower_priority": 0.0,
            "pci_scope_mentioned": 0.0,
            "deploy_windows_in_plan": 0.0,
            "major_version_flagged": 0.0,
            "executive_summary_exists": 0.0,
        }

    # Read available content
    triage_content = triage_file.read_text() if triage_file.exists() else ""
    remediation_content = remediation_file.read_text() if remediation_file.exists() else ""
    all_content = triage_content + "\n" + remediation_content
    all_lower = all_content.lower()
    triage_lower = triage_content.lower()

    # --- Check 3: All 10 CVEs covered in triage report ---
    cve_ids = [
        "CVE-2026-29112", "CVE-2026-31845", "CVE-2026-28003",
        "CVE-2026-30291", "CVE-2026-27519", "CVE-2026-33010",
        "CVE-2026-34201", "CVE-2026-25877", "CVE-2026-36500",
        "CVE-2026-37188",
    ]
    found_cves = sum(1 for cve in cve_ids if cve.lower() in triage_lower)
    scores["all_cves_covered"] = found_cves / 10.0

    # --- Check 4: Priorities assigned (P0-P3 labels present) ---
    priority_matches = re.findall(r'\bP[0-3]\b', triage_content, re.IGNORECASE)
    if len(priority_matches) >= 10:
        scores["priorities_assigned"] = 1.0
    elif len(priority_matches) >= 7:
        scores["priorities_assigned"] = 0.75
    elif len(priority_matches) >= 4:
        scores["priorities_assigned"] = 0.5
    elif len(priority_matches) >= 1:
        scores["priorities_assigned"] = 0.25
    else:
        scores["priorities_assigned"] = 0.0

    # Helper: find priority near a CVE reference
    def find_priority_near_cve(cve_id, content):
        """Look for P0-P3 within 300 chars of a CVE mention."""
        matches = list(re.finditer(re.escape(cve_id.lower()), content.lower()))
        if not matches:
            # Try package name as fallback
            return None
        for m in matches:
            start = max(0, m.start() - 150)
            end = min(len(content), m.end() + 300)
            section = content[start:end]
            p_match = re.search(r'\bP([0-3])\b', section, re.IGNORECASE)
            if p_match:
                return int(p_match.group(1))
        return None

    # --- Check 5: Express RCE is P0 ---
    express_p = find_priority_near_cve("CVE-2026-29112", triage_content)
    if express_p == 0:
        scores["express_rce_is_p0"] = 1.0
    elif express_p == 1:
        scores["express_rce_is_p0"] = 0.5
    else:
        scores["express_rce_is_p0"] = 0.0

    # --- Check 6: JWT bypass is P0 ---
    jwt_p = find_priority_near_cve("CVE-2026-31845", triage_content)
    if jwt_p == 0:
        scores["jwt_bypass_is_p0"] = 1.0
    elif jwt_p == 1:
        scores["jwt_bypass_is_p0"] = 0.5
    else:
        scores["jwt_bypass_is_p0"] = 0.0

    # --- Check 7: Build-only CVEs lower priority than runtime CVEs ---
    minimatch_p = find_priority_near_cve("CVE-2026-33010", triage_content)
    semver_p = find_priority_near_cve("CVE-2026-34201", triage_content)
    # Compare against express RCE priority
    if minimatch_p is not None and semver_p is not None and express_p is not None:
        if minimatch_p > express_p and semver_p > express_p:
            scores["build_only_lower_priority"] = 1.0
        elif minimatch_p > express_p or semver_p > express_p:
            scores["build_only_lower_priority"] = 0.5
        else:
            scores["build_only_lower_priority"] = 0.0
    elif minimatch_p is not None or semver_p is not None:
        # Partial credit: at least one build CVE is present and classified
        build_p = minimatch_p if minimatch_p is not None else semver_p
        if build_p is not None and build_p >= 2:
            scores["build_only_lower_priority"] = 0.75
        else:
            scores["build_only_lower_priority"] = 0.25
    else:
        scores["build_only_lower_priority"] = 0.0

    # --- Check 8: PCI DSS scope mentioned ---
    pci_patterns = [r"pci", r"payment card", r"pci.?dss", r"cardholder"]
    pci_found = any(re.search(p, all_lower) for p in pci_patterns)
    scores["pci_scope_mentioned"] = 1.0 if pci_found else 0.0

    # --- Check 9: Deploy windows in remediation plan ---
    remediation_lower = remediation_content.lower()
    deploy_patterns = [r"april\s*14", r"april\s*17", r"4/14", r"4/17", r"04-14", r"04-17"]
    deploy_found = sum(1 for p in deploy_patterns if re.search(p, remediation_lower))
    if deploy_found >= 2:
        scores["deploy_windows_in_plan"] = 1.0
    elif deploy_found >= 1:
        scores["deploy_windows_in_plan"] = 0.5
    else:
        scores["deploy_windows_in_plan"] = 0.0

    # --- Check 10: Major version upgrades flagged ---
    major_version_keywords = [
        r"major version", r"breaking change", r"major upgrade",
        r"8\s*(?:to|→|->)\s*9", r"4\s*(?:to|→|->)\s*7",
        r"2\s*(?:to|→|->)\s*4", r"major bump",
    ]
    major_found = any(re.search(p, all_lower) for p in major_version_keywords)
    # Also check for general upgrade risk language near relevant packages
    if not major_found:
        risk_keywords = [r"test.*(?:jsonwebtoken|helmet|debug)", r"(?:jsonwebtoken|helmet|debug).*test"]
        major_found = any(re.search(p, all_lower) for p in risk_keywords)
    scores["major_version_flagged"] = 1.0 if major_found else 0.0

    # --- Check 11: Executive summary in remediation plan ---
    first_chunk = remediation_lower[:max(len(remediation_lower) // 5, 300)]
    if re.search(r"(executive summary|summary|overview|tldr|tl;dr)", first_chunk):
        scores["executive_summary_exists"] = 1.0
    elif re.search(r"(executive summary|summary|overview)", remediation_lower):
        scores["executive_summary_exists"] = 0.5
    else:
        scores["executive_summary_exists"] = 0.0

    return scores

LLM Judge Rubric

Criterion 1: Priority Assignment Accuracy (Weight: 30%)

Score 1.0: All priorities are well-justified and contextually appropriate. Both CRITICAL CVEs (express RCE and JWT bypass) are P0 — especially the JWT bypass given active exploitation. The pg SQL injection is P0 or P1 with clear reasoning about PCI scope. High-severity runtime CVEs (lodash, axios) are P1. The helmet CSP bypass is P2 as a runtime medium. Build-only CVEs (minimatch, semver) are P2 or P3. Low-severity transitive dependencies are P3. Justifications reference both CVSS and deployment context (e.g., WAF mitigations, build-only vs runtime, exploit availability).

Score 0.75: Most priorities are correct with 1-2 minor discrepancies. All critical CVEs are P0. May slightly over- or under-prioritize one medium-severity CVE. Justifications are present but may miss some contextual factors.

Score 0.5: Gets the obvious critical CVEs right but struggles with the middle tier. May not differentiate between build-only and runtime vulnerabilities, or may ignore exploit availability when assigning priorities. Justifications are generic or missing.

Score 0.25: Significant misrankings. May assign priorities based purely on CVSS without considering deployment context. Critical CVEs may not all be P0.

Score 0.0: No priorities assigned, or priorities bear no relationship to severity and context.

Criterion 2: Contextual Security Reasoning (Weight: 25%)

Score 1.0: Demonstrates strong security judgment throughout. Recognizes that JWT bypass with active exploitation is the highest-urgency item. Identifies PCI DSS implications for the pg SQL injection and JWT bypass (affecting payment and auth flows). Notes that WAF provides partial mitigation for the SSRF but not full protection. Distinguishes build-time vs runtime exposure. Acknowledges that the express RCE having a public PoC increases urgency. Notes that low-severity transitive dependencies have conditional exploitability.

Score 0.75: Makes most key security observations. Identifies PCI scope for at least one relevant CVE. Distinguishes build vs runtime. May miss one or two contextual factors like WAF mitigation or the significance of active exploitation.

Score 0.5: Some contextual reasoning present but incomplete. May mention PCI in passing without connecting it to specific CVEs. Limited differentiation between build and runtime vulnerabilities.

Score 0.25: Minimal contextual reasoning. Treats the scan as a flat list without applying deployment context. Does not reference PCI, WAF, or exploit availability meaningfully.

Score 0.0: No contextual reasoning. Simply restates CVSS scores without analysis.

Criterion 3: Remediation Plan Quality (Weight: 25%)

Score 1.0: Remediation plan is actionable and operationally sound. Groups patches into logical work items (e.g., "critical hotfix for April 14" and "high-priority batch for April 17"). Maps specific CVEs to specific deploy windows. Identifies which upgrades are major version bumps and require additional testing (jsonwebtoken 8→9, helmet 4→7). Considers team capacity and sprint context. Suggests testing strategies for breaking changes. Executive summary is concise and suitable for leadership.

Score 0.75: Good plan with clear timeline and grouping. Covers both deploy windows. May miss one major version flagging or have slightly less specific testing guidance. Executive summary present and adequate.

Score 0.5: Basic plan exists with some timeline. May not effectively group patches or may not reference specific deploy windows. Limited attention to major version risks. Executive summary may be missing or too detailed.

Score 0.25: Plan is vague or generic. Lists CVEs to fix without actionable sequencing, timelines, or risk notes. No executive summary.

Score 0.0: No remediation plan or plan is unusable.

Criterion 4: Report Completeness and Structure (Weight: 20%)

Score 1.0: Both deliverables are complete, well-structured, and professional. Triage report covers all 10 CVEs with consistent formatting (CVE ID, package, CVSS, priority, justification, PCI scope). Remediation plan has clear sections (executive summary, timeline, risk notes, testing requirements). Content is scannable with good use of headers, tables, or bullet points. No CVEs omitted.

Score 0.75: Both files created with good structure. Minor formatting inconsistencies or one CVE with incomplete details. Generally professional and readable.

Score 0.5: Files created but structure is inconsistent. May be missing fields for some CVEs or lacks clear organization. Readable but requires effort to extract key information.

Score 0.25: Files created but poorly organized. Missing significant content for multiple CVEs. Hard to use as a working document.

Score 0.0: One or both files missing, or content is minimal/unusable.

Additional Notes

Embedded test signals in the vulnerability data:

Major version upgrades requiring extra testing:

• jsonwebtoken 8.5.1 → 9.0.1 (major)
• helmet 4.6.0 → 7.1.0 (major, skips multiple majors)
• debug 2.6.9 → 4.3.5 (major)
• qs 6.5.2 → 6.12.1 (minor but large gap)

Intentional design choices:

• The scan includes a mix of direct and transitive dependencies to test whether the agent distinguishes impact scope
• Build-only vulnerabilities are included to test whether the agent applies a different priority than runtime ones with the same CVSS
• The deployment context document provides explicit SLA timelines (24h/7d/30d/90d) that map to P0-P3, testing whether the agent uses this information
• PCI DSS scope is mentioned in the context to test whether the agent connects it to relevant CVEs (pg/SQL injection affecting database, JWT affecting authentication)
• The SOC 2 audit deadline (6 weeks) adds pressure context for the remediation timeline