A proposed Codex harness for ViralBench

ViralBench is a live agent environment

The live experiment described by the published methodology compares GPT-5.5, Claude Opus 4.8, and Kimi. Each receives two warmed-up TikTok accounts and runs around 7 AM and 5 PM Central Time, pursuing views in the fitness category.¹ These configuration details describe the reviewed version. The competition may since have changed.

I reviewed the 937-line TypeScript handoff. It contains the prompt and agent loop, with API clients and media handling. The same file also handles account telemetry, memory, rendering, publishing, and run persistence.² Unlike the two-account live competition, it uses one account and leaves a reviewable draft by default instead of scheduling publication.

The agent has up to 18 rounds and five tools:

TikTok and Instagram URLs resolve to images, which are resized, labeled, and returned to the vision-capable model. Generated images also receive labels and can guide later generations, giving the model visual working memory within a run.

Each run allows six Lightreel research calls, at most one per round, up to 12 viewed reference images per round, and 18 working rounds. OpenRouter calls the main model with an append-only conversation history. Same-round actions execute concurrently.

The workflow:

Retrieved examples may be unrepresentative, and TikTok’s distribution mechanism is unobserved. With few posts, the agent must infer whether a format failed or simply received weak distribution.

The environment combines partial observability, limited actions, changing conditions, and delayed rewards.

What the current code gets right

External research grounds the agent in recent content, while a finite budget forces it to choose questions.

The model sees source posts, generated images, and composed slides. Its visual review can therefore assess the artifact directly.

The single-file handoff makes tools, limits, failures, and memory easy to inspect.

Views are imperfect evidence, but the response comes from the platform.

The public handoff stops at a draft, preserving human review even though the live benchmark can publish autonomously. I would keep that boundary for company or client accounts.

The leaderboard compares configured systems

A post’s result depends on:

result = f(
  language model,
  system prompt,
  tool definitions,
  retrieval quality,
  reference-content selection,
  image model,
  account history,
  memory,
  posting time,
  publishing infrastructure,
  current TikTok conditions,
  random distribution effects
)

The model name is one input. The methodology aims to measure attention capture, but attributing a result to a model trait needs a defined measure and comparison. I would first judge the published configuration and its external outcome.

A whole-system comparison shows what one arrangement of model, account, tools, and budget achieved. The stronger question is why the outcome changed. A new image model, an account with different history and a favorable posting slot can all move the result while leaving the supposed reasoning improvement untested.

For an engineering decision, start with the mechanism the patch is meant to change. A preview gate should prevent unreviewed payloads. A memory change should retrieve relevant prior evidence without carrying a disproved belief forward. A new creative strategy must then survive a live comparison. These claims can share a run record, but they should not share a success label that lets one good views result excuse every other failure.

A weekly winner is therefore a candidate for investigation. Promotion needs a versioned configuration, a named effect and evidence that could distinguish that effect from account history, imitation and distribution noise.

What the code audit reveals

The handoff makes the content loop inspectable. Turning it into an improvement platform requires records that can explain a result and checks that can reject an invalid one.

1. The run record is too thin to explain success or failure

The local .marketing-agent-runs.json records the goal, published posts, final answer, reflection note, and round count.

It omits the trace and the lineage needed to reconstruct the run. The missing fields fall into four groups:

The stored summary cannot reliably tell whether retrieval, generation, publishing or the model caused a failure. It also cannot explain which decisions contributed to a successful post.

Full traces should be the improvement harness’s primary dataset.

2. Several important constraints exist only as prompt prose

The prompt requires previewing before publication, a unique image per slide, exactly one post per account, and completion before the final answer.

The runtime enforces maximum posts per account but accepts a final answer without checking the minimum. It requires no successful preview token, rejects no duplicate image URLs, and does not validate slide count, copy length, text fit, unsupported claims, or generation lineage.

Prompts influence behavior. Code must enforce these invariants.

3. Visual review is incomplete for longer carousels

Preview renders at most six slides. A longer carousel can reach publication with its final slides unseen.

Render every slide or block publication. Bind the preview receipt to immutable asset hashes, slide order, text, styling, and renderer version, then make the gateway reject a publication whose content differs from that receipt even if every image URL still looks identical, because those addresses can serve changed bytes after approval. The payload changed. Preview it again.

4. Cross-run memory is compressed into free-form notes

Future runs load up to ten model-written reflection notes as their only durable strategic memory.

These notes mix facts, interpretations, and narrative without checking them against the trace. They provide no stable representation of tested hypotheses, confidence, failed formats, account-specific effects, or unresolved questions.

The model writes its memory. The implementation still needs a checker that compares each claim with its trace.

5. Account telemetry has attribution edge cases

The code refreshes recent posts’ TikTok metrics through ScrapeCreators when a public URL exists. Without one, it matches a profile post using a normalized caption prefix.

Two posts can share a normalized caption prefix while belonging to different experiments, so the ledger needs a stable post ID tied to the publishing response and each later metrics snapshot before a dashboard can attribute the views to the candidate that produced them. Similar words are weak identity.

A publishing timeout creates a related problem: the platform may have accepted the draft even though the caller did not receive the response. The proposed adapter should reconcile a stable operation or draft ID before retrying, using provider idempotency support where available. An uncertain result should stay unresolved until reconciliation can rule out a duplicate submission.

6. Same-round parallelism creates hidden execution semantics

Promise.all runs same-round actions concurrently. This suits independent actions, but one image generation cannot reliably use another’s new label in that round. It must wait until a later round.

The prompt leaves this dependency implicit. A typed planner or action graph would express it.

7. The content-acquisition strategy creates provenance risk

The prompt asks the agent to adapt successful posts’ language closely and sometimes mirror their slide sequence. That may help short-term results, but it conflates:

• learning a general format;
• adapting a common hook pattern;
• paraphrasing a specific creator’s copy;
• reproducing a distinctive sequence or visual treatment.

The code neither stores a source graph nor measures reference-to-artifact similarity.

The agent should learn why a post works without reproducing a creator’s distinctive work.

8. The package is not fully reproducible as shipped

The source imports sharp for resizing and instructs users to install it, but package.json omits the dependency.

Unattended tests need a clean installation. The manifest must declare what the code imports.

What I mean by “Codex as a harness”

OpenAI describes the Codex harness as the loop and execution logic connecting a model to instructions, tools, filesystem operations, and feedback.³ Its harness-engineering work also includes the repository, documentation, tests, review process, execution environment, and validation rules that govern reliable work.⁴

In this design, ViralBench is the executable marketing environment. Codex works on it from outside the content-production loop.

I would assign these roles:

Keep the evaluator, success criteria, and release authority outside the patching agent’s control. Otherwise it could improve its score by weakening the test.

Freeze development cases and a promotion set the patching agent has not seen, then govern what feedback promotion reveals. Keeping the grader outside Codex does not preserve statistical independence if the team repeatedly chooses patches after seeing the same holdout scores. Dwork and colleagues’ work on adaptive data analysis explains why reuse can turn validation data into another source of tuning. Adaptive holdout reuse.

I would record each promotion query, limit the feedback exposed during development and reserve fresh cases for the final candidate. A repaired failure can become a regression fixture; it should not simultaneously serve as untouched evidence that the repair generalizes. This conservative workflow does not implement the paper’s reusable-holdout algorithms or claim their formal guarantees.

I would assign five functions:

1. The inner agent researches, creates, previews, and submits a post.
2. The recorder captures raw evidence, actions, artifacts, and outcomes.
3. The evaluator applies versioned checks and statistical rules.
4. Codex diagnoses a bounded issue and produces a patch in an isolated worktree.
5. The release controller decides whether the candidate can enter replay, draft canary, or live deployment.

The proposed control flow:

                  LIVE ENVIRONMENT
      Lightreel / TikTok / Doublespeed / media APIs
                         ↑       ↓
                ┌───────────────────┐
                │ ViralBench agent  │
                │   inner loop      │
                └───────────────────┘
                         ↓
          immutable traces, sources, artifacts,
            configuration, cost, metric snapshots
                         ↓
                ┌───────────────────┐
                │ Evaluation layer  │
                │ checks + analysis │
                └───────────────────┘
                         ↓
              bounded experiment candidate
                         ↓
                ┌───────────────────┐
                │ Codex worktree    │
                │ plan / patch/test │
                └───────────────────┘
                         ↓
             offline replay and regression gate
                         ↓
               human-approved draft canary
                         ↓
                 locked online evaluator
                         ↓
                   promote or revert

The trace has to explain the grade

Keep observations in their own records before deriving judgments.

Responses, rendered pages, schema, and search results are observations. Scores and recommendations interpret them. A revised judgment must not erase the evidence it used.

For ViralBench, each run should preserve an immutable evidence bundle:

Store derived judgments in their own records:

• “insufficient research breadth”;
• “slide 1 text is likely unreadable”;
• “final copy is too similar to source post X”;
• “the treatment improved account-normalized 72-hour views”;
• “the agent repeated a failed format despite contrary evidence.”

Each finding should cite its supporting evidence IDs. Codex may propose findings and patches; it must not rewrite the raw trace.

Complete records let reviewers compare a model’s explanation with the sequence of events.

A task, a trial and a grade need their own identities

The task defines the initial fixture, allowed actions, stopping conditions and success contract. A trial is one attempt by one versioned model-plus-harness configuration. Its trace orders the observable events; a grade applies a versioned check or rubric to the resulting evidence. Changing the fixture or criterion changes the task version, while another stochastic attempt gets another trial ID.¹²

Use harness-visible evidence: instructions, responses, tool contracts, calls and results, artifact hashes and state changes. Hidden chain-of-thought is unnecessary. Sensitive trace bodies need redaction or controlled references before they enter general analytics. Durable run and artifact IDs should survive asynchronous work; a sampled operational trace is helpful correlation, but may not keep the evidence needed to reproduce a grade.¹⁴

This illustrative manifest joins those records. Its IDs, cost, and duration are invented values for the example. No ViralBench run is reported here:

{
  "trial_id": "trial-0184",
  "task": { "id": "publish-draft", "version": "3" },
  "configuration": { "id": "agent-candidate", "version": "7" },
  "environment": { "fixture": "workspace@sha256:...", "network": "recorded" },
  "trace": { "id": "trace-0184", "integrity": "complete" },
  "graders": ["outcome@2", "policy@5", "side-effects@3"],
  "result": { "outcome": "pass", "policy": "pass", "side_effects": "pass" },
  "cost_usd": 0.42,
  "duration_ms": 18420
}

A trial must also leave a known environment. Snapshot state before and after, attribute mutations to events, and restore or discard the isolated namespace after failure. A draft or partial write left behind can contaminate the next comparison even when the agent’s final message says it failed cleanly.

The experiment is the unit of change

An experiment needs a falsifiable hypothesis and protected variables before Codex edits.

An example:

experiment_id: visual-critic-v1
hypothesis: >
  A separate visual-critique pass after rendering will reduce
  text-overflow, repeated-image, and synthetic-looking-image failures
  without lowering 72-hour account-normalized views.

baseline_commit: 5f5f57e251023ceb37961c0fc2c808f67ceb71eb
editable_surfaces:
  - agent/critics/visual.ts
  - evals/visual/*
  - experiments/visual-critic-v1.yaml

protected_surfaces:
  - evals/online/primary-metric.ts
  - policies/publish-gateway.ts
  - traces/raw/*
  - secrets/*

offline_gates:
  - all_slides_rendered
  - no_duplicate_images
  - no_text_overflow
  - source_lineage_complete
  - historical_replay_pass_rate >= 0.95

live_design:
  unit: post
  blocking:
    - account
    - posting_slot
    - calendar_week
  primary_window: 72h
  minimum_pairs: 20

primary_metric: account_adjusted_log_views_72h
guardrails:
  - publish_failure_rate <= baseline
  - median_cost_per_post <= baseline * 1.15
  - policy_violation_count == 0
  - source_similarity_high_risk_count == 0

promotion_rule: >
  Pass every hard guardrail and show a positive primary-metric estimate
  with the pre-registered uncertainty threshold.

Define the hypothesis, editable scope, validation commands, metrics, guardrails, and stopping rule before Codex edits code or anyone sees live results. The example’s minimum_pairs: 20 and 0.95 replay threshold are illustrative planning values. They have not been validated through a power calculation or release testing. A real manifest needs thresholds justified by baseline variance, minimum useful effect and the cost of a mistaken promotion.

Save rejected experiments too: their traces explain which hypothesis failed and why.

Offline replay should absorb most iteration

Replay should handle parser, tool, and prompt regressions before scarce accounts, posting slots, API budgets, and review time are committed to live trials.

Convert historical calls into versioned fixtures:

A historical trace is not a complete environment model. Match a frozen response to the request the candidate actually makes. A new question, account state or action branch can leave the recording without a valid answer; stop as unmatched or enter an explicitly modeled scenario. Feeding the next historical response to a different request creates an apparently successful trajectory by changing its world after the fact.

An unmatched request identifies the next fixture or observation needed. I would use replay to certify bounded invariants and expose known failures, then choose a live experiment for the creative questions it leaves open. The audience response to an unseen post remains outside the recording.

Deterministic checks

Check these failures without another model:

Rubric-based checks

Other qualities need judgment:

• Is the first slide visually legible at mobile size?
• Does the carousel maintain a coherent subject across slides?
• Does the copy preserve the mechanism of a reference without reproducing it?
• Does the post contain unsupported fitness or health claims?
• Does the final creative look internally consistent?

Give model-based graders the evidence and a narrow rubric. Their scores express evaluator judgments and require adjudication.

AgentRewardBench found substantial disagreement between automatic judges and expert annotations; its rule-based evaluators also missed successes when exact criteria were too narrow. Independence from the patching agent is therefore necessary for this design, but insufficient to make a grade correct. Calibrate on evidence-linked expert decisions and inspect both false passes and false rejections. Keep the model, prompt, rubric and input representation with each grade.¹³

For a claim about a rendered carousel, the final artifact matters. For a claim about valid tool use, the trace and resulting environment matter too. Record those results alongside authorization, side effects, efficiency, and integrity, with an outcome for each.¹⁶ A single aggregate grade can hide the reason the candidate should not advance.

Compare completion rates only across gradeable, in-scope trials. Missing traces or a broken fixture can leave a trial ungradeable; an out-of-scope action can make a nominal completion unacceptable. Consider an illustrative baseline completing 8 of 10 trials and a candidate completing 9 of 10, with one candidate success sending an unauthorized message. The unauthorized message fails the candidate regardless of its completion count. Report both outcomes and the small denominator, then inspect paired changes across the same task versions before promoting the configuration.¹⁵

Adversarial and fault-injection cases

Inject dependency failures:

The agent should stop safely and save a diagnostic trace.

OpenAI’s evaluation guidance combines prompts, captured runs, traces, artifacts, checks, and comparable scores.⁵ I would use that structure here.

Live evaluation needs experimental discipline

A replay pass qualifies a candidate for controlled live testing. Whether the change increases views remains a live-test question.

Views are noisy and path-dependent

TikTok outcomes depend on the account and the audience as well as the content. Timing and platform distribution add more variation. The live experiment should record these conditions:

The same creative cannot reach the same audience twice under identical conditions.

Posts can change the account’s audience and later distribution, so sequential outcomes are not independent observations.

Raw mean views can mislead with small samples and rare winners. Online business metrics often have skewed, heavy-tailed distributions; controlled experiments use robust estimators, covariate adjustment, and longitudinal methods to address them.⁶

The trial should be blocked and paired

With few accounts and slots, I would block concurrent trials against a baseline.

Pair each candidate with a concurrent baseline, controlling where possible for:

A switchback could randomize versions across comparable account-time blocks, but the carryover model is part of the design, not a nuisance to remove by alternating labels. Bojinov, Simchi-Levi and Zhao explicitly model the duration of effects from earlier treatments and discuss the information needed to choose it. If a post durably changes the account’s audience or later distribution, the available schedule may not support a credible short washout. The paper doesn’t show that TikTok meets its assumptions.¹¹

That can change the decision to run the trial. A parallel account-level design avoids switching one account back and forth, but needs enough independent accounts and a defensible assignment. Many posts on a few accounts do not create many independent account histories. If neither design is credible at the available budget, I would report the comparison as exploratory and prioritize the reliability fixes whose mechanism can be checked offline. An inconclusive creative experiment is preferable to a precise estimate for an effect the design never identified.

Pre-experiment performance could serve as a covariate, using variance-reduction methods such as CUPED to account for different starting levels.⁷ Use information fixed before treatment exposure, and keep the assignment mechanism and adjustment formula in the manifest. With few accounts, many posts are still repeated observations on those accounts; they do not create a large independent account sample.

Measurement windows should be fixed in advance

Capture views and engagement at fixed horizons, such as:

• one hour;
• six hours;
• 24 hours;
• 72 hours;
• seven days.

Choose the primary horizon before the trial. A favorable six-hour result cannot replace a planned 72-hour result.

I would analyze an account-adjusted log(1 + views) alongside the raw distribution. The log scale limits an extreme post’s influence; median and trimmed-mean comparisons would check robustness.

Show every raw outcome alongside robust estimates so readers can see the outliers’ influence. Improvement in mean log views is an effect on that transformed metric, not automatically the same percentage improvement in total raw views. Fix how account adjustment is computed before the trial, report both scales and match uncertainty calculations to the randomized block structure.

Hard guardrails should precede optimization

Evaluate policy, provenance, factuality, and disclosure on their own terms before considering views. A weighted score must not let high reach offset a failed constraint.

The proposed objective:

maximize expected account-adjusted views at the locked horizon

subject to:
| Item | Requirement or recorded value |
| --- | --- |
| 1 | platform and disclosure requirements pass |
| 2 | source-provenance checks pass |
| 3 | factual and health-claim checks pass |
| 4 | publishing and rollback checks pass |
| 5 | cost remains within budget |
| 6 | human approval is present where required |

Only candidates that pass these constraints compete on views.

A/B tests measure; bandits allocate

A multi-armed bandit could allocate posts to promising strategies. It cannot replace causal evaluation.

An A/B or switchback trial asks whether a change improved outcomes under comparable conditions.

A bandit chooses where to allocate the next unit of traffic or publishing capacity, balancing reward and learning.

Use controlled trials to measure effects, then bounded bandits to allocate among validated strategies. Mixing measurement and allocation too early obscures causality.

Codex should receive bounded engineering tasks

Codex can inspect, edit, and run code through its CLI, non-interactive codex exec, TypeScript SDK, or an orchestration layer.⁸ The SDK can start and resume coding threads programmatically. Worktrees give each branch its own checkout; their filesystem isolation does not itself restrict credentials, network access or shared services. The configured sandbox and tool permissions must enforce those restrictions.⁹ A structured final report helps integration by identifying the patch, checks and artifacts. An independent gate still has to inspect them before accepting its claims.

I would begin with codex exec in CI and a structured output schema. The SDK could later support persistent experiment threads and staged tasks.

Each task should define:

I plan to support these task modes:

“Deploy” would execute an approved release, with no authority to approve it.

A hypothetical handoff follows. Its VB-2026-07-08-017 trace ID and eight-slide incident are invented to show the contract; they are not an observed run. The commands are proposed targets that require corresponding fixtures and scripts before they can execute:

Mode: Implement
Goal: Make a successful full-slideshow preview a hard prerequisite for publishing.

Evidence:
- Trace VB-2026-07-08-017 published eight slides after previewing only six.
- The current preview function caps max_slides at six.
- publish_slideshow has no preview receipt check.

Editable paths:
- agent/rendering/**
- agent/publishing/**
- evals/preview-gate/**

Do not edit:
- evals/online/**
- policies/**
- traces/raw/**
- experiment promotion thresholds

Verification:
- npm test
- npm run eval:preview-gate
- npm run replay -- --case VB-2026-07-08-017

Done when:
- every slide is rendered or publication is blocked;
- a preview receipt is bound to the exact slide payload;
- the historical failure becomes a passing regression case;
- RESULTS.md contains commands, results, changed files, and remaining risks.

This handoff defines observable behavior, protected files, and acceptance checks.

The repository itself must become legible to Codex

Codex needs relevant knowledge in its repository. OpenAI’s harness-engineering work emphasizes that information confined to chats, external documents, or memory is unavailable to the agent while it works.⁴

I would split the handoff into modules and versioned artifacts:

/agent
  /policy              system prompts, action schemas, run policy
  /runtime             loop, state machine, dependency handling
  /memory              structured cross-run state

/tools
  /lightreel
  /media
  /images
  /rendering
  /publishing
  /metrics

/experiments
  /manifests            hypotheses, protected variables, promotion rules
  /candidates           Codex-ready tasks and results

/replay
  /fixtures             frozen external responses and assets
  /worlds               named historical environment states
  /faults               dependency and data failure scenarios

/evals
  /deterministic
  /rubrics
  /visual
  /online
  /regression

/traces
  /raw                  append-only event records
  /indexes              searchable run and artifact indexes

/artifacts
  /source-media
  /generated-media
  /previews
  /published

/reports
  /runs
  /experiments
  /leaderboard

/codex
  AGENTS.md
  /skills
  /schemas
  /hooks
  /task-templates

/policies
  publish-gateway.ts
  provenance.ts
  platform-compliance.ts
  health-claims.ts
  cost-limits.ts

AGENTS.md should point to architecture, evidence contracts, task modes, checks, and protected paths. Detailed decisions belong beside the code they govern, in versioned documents and schemas.

Hooks could require an experiment ID, protect restricted files, validate result-to-trace references, and confirm required evaluations before accepting a pull request. Code would enforce the task contract.

Memory needs evidence, scope, and an expiry date

The existing reflection note could remain, but structured memory should record:

An example:

{
  "belief_id": "BEL-0041",
  "account": "fitness-account-a",
  "claim": "Beginner mistake carousels outperform motivation quotes in the evening slot",
  "evidence_runs": ["RUN-0108", "RUN-0112", "RUN-0117"],
  "counterevidence_runs": ["RUN-0120"],
  "confidence": 0.63,
  "valid_from": "2026-07-01",
  "review_after": "2026-07-15",
  "status": "provisional"
}

The system could derive provisional beliefs from traces while keeping narrative notes. Every evidence reference must resolve.

Expire or discount beliefs when their evidence ages, the account changes, or results contradict them.

Keep a small, contestable set of beliefs. Repetition must not become evidence.

Each slide needs a path back to its sources

The audited prompt encourages close adaptation, making source lineage an early requirement.

Record each final slide’s lineage:

“Problem, explanation, correction, payoff” is a common pattern. A creator’s distinctive sentences, examples, and compositions may be reproduction. The evaluator should flag possible reproduction for review.

Under the TikTok guidance cited in July, realistic AI-generated images, audio, and video require labels; Content Credentials may also trigger labeling.¹⁰ Because ViralBench seeks realistic imagery, the proposed gateway should keep disclosure metadata and block publication when a required setting cannot be verified.

A views-only objective may reward extreme nutrition claims, impossible transformations, or fear-based advice. Claims about health, injury, supplementation, or disease need their own evidence and policy check before publication.

Views measure attention; revenue needs another test

The current goal makes views the sole objective, excluding brand safety, selling, and follower count.

Views provide an observable target.

It may also favor copied hooks, outrage, exaggerated promises, high-arousal content, or account tactics that sacrifice future performance.

I would use two modes.

Benchmark mode

Study attention capture under controlled conditions.

Optimize account-adjusted views subject to platform policy, disclosure, provenance, factuality, cost, and publishing constraints.

Enterprise mode

Measure business outcomes:

Business metrics have different horizons and causal roles. Report leading indicators alongside outcomes, keeping attribution uncertainty visible.

Commercial value requires evidence about customers and purchases.

ViralBench can test attention capture. Connecting it to revenue requires downstream measurement and appropriate constraints.

The first build sequence

Make the baseline reproducible and observable before changing prompts.

Phase 0: Freeze and document the baseline

• Pin the current commit and dependency versions.
• Add the missing sharp dependency and a lockfile.
• Record the exact live configuration and handoff configuration as named versions.
• Create an architecture map and a canonical run schema.
• Save a small set of representative historical runs as reference cases.

Deliverable: an installable, replayable baseline.

Phase 1: Add complete traces and hard runtime invariants

Deliverable: traces that explain each run.

Phase 2: Split the monolith and build replay

• Extract provider and tool adapters behind typed interfaces.
• Add mock implementations and frozen fixtures.
• Build deterministic checks and regression cases.
• Add fault injection for each external dependency.
• Preserve the original single-file behavior as a compatibility baseline.

Deliverable: local tests that require no publication.

Phase 3: Introduce Codex as a reviewed engineering agent

• Add AGENTS.md, task templates, output schemas, and protected-path rules.
• Run Codex in isolated worktrees with workspace-scoped permissions.
• Start with Plan, Review, Debug, and Implement modes.
• Require a human to approve every patch and experiment manifest.
• Store Codex transcripts, diffs, test results, and review decisions.

Deliverable: reviewed engineering patches.

Phase 4: Add controlled draft canaries

• Allow a passing candidate to create a draft on a designated test account.
• Run visual, provenance, claim, and disclosure checks on the final artifact.
• Require human approval to publish.
• Collect fixed-window outcomes through the locked evaluator.
• Compare treatment against a concurrent blocked baseline.

Deliverable: a controlled online trial with a rollback point.

Phase 5: Automate promotion for a defined class of changes

Automate low-risk promotion only after offline and online gates demonstrate reliability.

Early candidates could include:

Keep these changes under review:

Expand autonomy only where reliability has been demonstrated.

The experiments I would run first

With traces and replay in place, I would test:

Does a second visual critic improve the final artifact?

A second critic may catch repeated images, inconsistent people, overflow, or artificial compositions. It may also increase cost and over-edit good work.

Test visible defects offline before publication.

Does structured memory outperform free-form notes?

Replace or supplement the ten notes with evidence-linked beliefs, account-specific hypotheses, and expiry dates.

Replay would test relevant memory use and rejection of stale or contradicted beliefs. Live trials would test account-adjusted performance consistency.

Is candidate generation better than single-path creation?

Instead of committing to one strategy, generate several cheap concepts, score them against current evidence, and spend image-generation budget on the strongest.

A critic may favor consensus and reject unusual, promising ideas. Measure diversity, cost, and live outcomes.

How much research is enough?

The six-call Lightreel budget has not been shown to be optimal. Compare four, six, and eight calls on source breadth, decisions, cost, and views.

Query diversity, full-post inspection, or a copy review against evidence may matter more than call count.

Should planning and execution use different models?

One model could choose the thesis and sources; another could create and inspect the post. Specialization may help, but handoffs may lose information.

Compare system outcomes and trace quality across these configurations.

Does source similarity predict either performance or risk?

Source lineage would let us test the prompt’s assumption that close adaptation improves views. Compare originality bands on views, reviewer concern, and platform outcomes.

This could locate where learning a format stops helping and duplication risk begins.

What this becomes beyond TikTok

A TikTok carousel is the first environment. The proposed engineering loop could serve other marketing systems:

• SEO and AEO recommendations generated from public-web evidence;
• paid-media agents that propose and test campaign changes;
• sales-content agents that build materials from product and customer evidence;
• brand-monitoring agents that investigate emerging narratives;
• content operations that generate, review, publish, and learn across channels.

Each would need to:

The shared principle is to keep evidence inspectable, tie derived claims to it, and validate proposed actions independently.

The durable asset is the learning system

The first result may be a narrow one: a candidate can no longer publish a payload whose last slides were unseen, and an independent check confirms that property under the failure cases that used to break it. That is progress even before a new views record appears.

The harder claim is that the system learns to make better creative decisions. It requires more than a trace archive, an agreeable judge and repeated attempts on familiar accounts. The promotion evidence has to survive adaptive tuning and an experimental design appropriate to the platform’s carryover. Where those conditions cannot be met, the result should stay exploratory.

I would build the smallest outer loop that can expose a mistaken improvement claim. If it improves reliability at an acceptable cost, keep it. If a simpler replay suite and human review do that job as well, use them until the creative experiments justify more machinery. The durable asset is the set of decisions that survived an honest opportunity to fail, with enough evidence for someone else to challenge them.

Sources and technical notes

This article draws on the live methodology and a supplied audit of repository commit 5f5f57e251023ceb37961c0fc2c808f67ceb71eb.

1. ViralBench methodology and live experiment. ↩
2. ViralBench repository and the audited marketing-agent.ts commit. ↩
3. OpenAI, “Unrolling the Codex agent loop”, January 23, 2026. ↩
4. OpenAI, “Harness engineering: leveraging Codex in an agent-first world”, February 11, 2026. ↩
5. OpenAI Developers, “Testing Agent Skills Systematically with Evals” and “Build an Agent Improvement Loop with Traces, Evals, and Codex”. ↩
6. Chen Wei et al., “Beyond Basic A/B Testing: Improving Statistical Efficiency for Business Growth”, revised December 27, 2025. ↩
7. Microsoft Research, “Deep Dive Into Variance Reduction”. ↩
8. OpenAI Developers, Codex SDK and Codex non-interactive mode. ↩
9. OpenAI Developers, Codex sandboxing and Codex worktrees. ↩
10. TikTok Support, “About AI-generated content”. ↩
11. Iavor Bojinov, David Simchi-Levi and Jinglong Zhao, “Design and Analysis of Switchback Experiments”, revised September 17, 2025. Used here for randomization and carryover assumptions; no ViralBench trial is reported. ↩
12. Anthropic Engineering: demystifying evals for AI agents. ↩
13. AgentRewardBench: evaluating automatic evaluations of web-agent trajectories. ↩
14. W3C Trace Context Recommendation. ↩
15. Anthropic: a statistical approach to model evaluations. ↩
16. OpenAI API reference: Evals. ↩

The AI-manager cases test a related operational question: whether errors change the next shift’s behavior when memory, permissions and human intervention are part of the system.