How to run continuous agentic red teaming against your own apps

Quick answer

Write a scope file that lists the staging hosts and APIs you own and authorize for testing, give the agent a rules-of-engagement skill, and schedule a nightly Nori automation. Each night the agent retests open findings, probes what changed, and confirms every new issue with a minimal reproduction. Confirmed findings land in Slack and Linear with reproduction steps, request logs and severity, and code fixes arrive as draft pull requests for a person to review.

What continuous agentic red teaming is

A yearly pen test checks the application you had on the day of the test. Your team ships new endpoints every week after that. Continuous agentic red teaming closes the gap: an AI agent tests your own applications every night, inside a written scope, and reports only what it can reproduce.

On Nori, the red team is a scheduled session. A cron automation starts an isolated cloud sandbox with your repositories cloned and your security skillset installed. The agent reads the scope, does the work with ordinary tools (curl, an HTTP proxy, a scanner, the application's own test suite), posts its results in a Slack thread, and the sandbox is released.

Three things separate this from a vulnerability scanner on a timer:

• It reads your code. The agent has the repository, so it reads the route handlers and the authorization checks, then tests the ones that look wrong. It also reads last night's merged pull requests and tests the new endpoints first.
• It confirms before it reports. A finding needs a request and a response that prove it. Scanner output that the agent cannot reproduce goes in a short "not confirmed" list, not in a ticket.
• It follows through. A finding becomes a ticket, often a draft fix pull request, and a retest on the next run.

The safety rules the agent works under

• Your own systems only, listed by name in a scope file with a named owner and an expiry date.
• Staging first. Add production later, host by host, by an approved pull request.
• Rate limits per host, and a stop condition when the target starts to return errors.
• No destructive actions: no denial of service, no deleting data, no attacks on real user accounts, no social engineering.
• Stop and ask when the next step would touch anything outside scope. A person answers in Slack.
• Never merge. Fixes are draft pull requests.

These rules live in a rules-of-engagement skill that every security session loads. The full skill is in the 24/7 red and blue team setup. The method itself follows public testing guides such as the OWASP Web Security Testing Guide and NIST SP 800-115.

How to run it, step by step

1. Write the scope file. Commit security/scope.yaml to a repository the agent can read. Require the security lead's review on every change.
2. Create test accounts on staging. At least two tenants with two users each, so the agent can test whether one tenant can reach another's data.
3. Install the rules-of-engagement skill in a security-ops skillset and publish it to your Nori organization.
4. Connect the tracker. Add the Linear integration (or Jira, or GitHub Issues) and scope it to the security agent.
5. Create the nightly automation with the schedule below. Point it at a Slack channel so each run is a thread.
6. Run it once by hand. Use Run now, read the thread, and check the request count against your rate limits.
7. Triage the first findings together. Reply in the thread to correct severity or mark accepted risk. The agent saves those decisions to shared memories and applies them on later runs.
8. Review weekly. Look at open findings by age, and add newly shipped hosts to the scope file.

Example scope file and schedule

This scope covers one staging API and its web front end. The focus list tells the agent where to spend its time.

# security/scope.yaml
authorization:
  owner: head-of-security@example.com
  approved_on: 2026-11-02
  valid_until: 2027-02-01

in_scope:
  - name: staging-api
    url: https://api.staging.example.com
    openapi: services/api/openapi.yaml
    max_requests_per_second: 5
    test_accounts:
      tenant_a: [redteam-a1@example.com, redteam-a2@example.com]
      tenant_b: [redteam-b1@example.com]
    focus:
      - broken object-level authorization
      - authentication and session handling
      - injection in search and import endpoints
      - endpoints added in the last 7 days
  - name: staging-web
    url: https://app.staging.example.com
    max_requests_per_second: 3

out_of_scope:
  - production
  - third-party services (payments, email, identity provider)

forbidden_actions:
  - denial of service
  - modifying or deleting data outside the test tenants
  - credential attacks on real accounts

markers:
  request_header: 'X-Redteam-Run: <run id>'

The schedule is one Nori cron automation. It runs at 02:00 in your timezone, after the nightly staging deploy.

nori-sessions triggers create \
  --name 'Red team nightly' \
  --type cron --cron-schedule '0 2 * * *' --timezone America/New_York \
  --skillset security-ops \
  --slack-channel-id C0SECAGENTS \
  --prompt 'Follow the rules-of-engagement skill. 1) Retest every open Linear ticket in team SEC and update it. 2) List pull requests merged to example/api since the last run and test new or changed endpoints first. 3) Spend the remaining time on the focus list in security/scope.yaml. Report only findings you reproduced. For each: create a SEC ticket and, if the fix is in our code, open a draft pull request with a failing test. End with a summary.'

What a nightly run does

1. Check authorization

The agent reads the scope file, confirms the authorization has not expired, and resolves each hostname. A host that is not on the list is not tested.

2. Retest open findings

For each open ticket it replays the stored reproduction. If the request now fails as it should, the ticket gets a "retest passed" comment and moves to done. If it still works, the ticket gets the new evidence.

3. Test what changed

It reads the diff of each pull request merged since the last run. A new GET /api/v1/invoices/{id} handler without a tenant check is a lead. The agent then proves or disproves it against staging with the two test tenants.

4. Work the focus list

The rest of the run goes to the focus areas, mapped to OWASP testing categories. The agent searches shared memories first, so it does not repeat a test that was closed as a false positive.

5. Report

Confirmed findings go to Linear and to the Slack thread. Anything that needs a person, such as an unclear severity or a lead that points outside scope, is asked as a question in the thread.

Example finding

This is the ticket the agent writes. The data is illustrative. Every finding has the same fields, so a reviewer can reproduce it in a minute.

SEC-212  Invoice API returns other tenants' invoices
Severity: high        Target: staging-api        Found: 2026-11-11 02:31
Category: broken object-level authorization
Introduced by: example/api#1832 (merged 2026-11-10)

Reproduction
1. Log in as redteam-a1@example.com (tenant A). Keep the bearer token.
2. Log in as redteam-b1@example.com (tenant B). Create an invoice. Note its id: inv_8f31.
3. As tenant A, request the tenant B invoice:

   curl -s https://api.staging.example.com/api/v1/invoices/inv_8f31 \
     -H 'Authorization: Bearer <tenant A token>' \
     -H 'X-Redteam-Run: rt-2026-11-11'

Expected: 404
Actual:   200 with the full invoice body of tenant B

Evidence
  request log: runs/rt-2026-11-11/requests.jsonl, lines 388-391
  response excerpt: {"id":"inv_8f31","tenant_id":"tnt_b","total":120.00}

Cause
  services/api/invoices/handlers.py get_invoice() loads by id only.
  list_invoices() filters by tenant_id; get_invoice() does not.

Suggested fix
  Filter by request.tenant_id in get_invoice(). Draft PR: example/api#1841
  The PR adds test_get_invoice_other_tenant_returns_404 (fails before, passes after).

Retest: scheduled for the next nightly run.

The Slack thread carries the short form:

Red team nightly  02:00
Tested 2 targets, 388 requests, peak 4 requests per second.
NEW   SEC-212 (high) Invoice API returns other tenants' invoices. Fix PR: example/api#1841 (draft)
PASS  SEC-198 Missing rate limit on password reset. Closed.
OPEN  SEC-205 Session cookie without Secure flag. Still reproduces, 9 days old.
Not confirmed: 2 scanner hits on /api/v1/search, could not reproduce.

Fixes and retests

The draft pull request is a normal pull request. A code owner reviews it, asks for changes in a comment, and the agent pushes a follow-up commit. Nobody has to open the session; the review happens where your team already reviews code.

Once the fix is merged and deployed to staging, the next nightly run replays the reproduction. A passing retest closes the ticket with the evidence attached. A failing retest reopens it and says so in the thread. This is the part a yearly test cannot give you: every fix is checked within a day.

The same traffic is useful to your defenders. Because every request carries the marker header, the blue-team triage agent can check which red-team techniques raised an alert and which went unseen.

What this does not replace

Agents complement a certified pen test; they do not replace one where compliance requires it. An independent human tester brings judgment about business logic, chained attacks and physical or social paths that are out of bounds for an agent, and produces the signed report that auditors and customers ask for. Use the agents for nightly coverage and retests, and book the pen test on its usual cycle. The trade-offs are laid out in agentic red teaming vs pen tests vs BAS.

Frequently asked questions