[draft] [ don't merge] Add negative-case translation repro for the eligible-gold action set

[Dominic Nguyen (datduyng)]

Full run

negatives=20 models=sol concurrency=4 (real TypeAgent translator)

FAIL BAD_NEGATIVE sol   | target=browser.goForward
   utterance:       Do not navigate forward in the browser.
   expectedActions: []
   produced:        ["browser.goBack"]

FAIL BAD_NEGATIVE sol   | target=browser.goBack
   utterance:       Don't go back to the previous page.
   expectedActions: []
   produced:        ["browser.goForward"]

FAIL BAD_NEGATIVE sol   | target=browser.scrollDown
   utterance:       Do not scroll down this page.
   expectedActions: []
   produced:        ["browser.scrollUp"]

FAIL BAD_NEGATIVE sol   | target=browser.scrollUp
   utterance:       Please don't scroll up.
   expectedActions: []
   produced:        ["browser.scrollDown"]

FAIL BAD_NEGATIVE sol   | target=browser.stopReadPageContent
   utterance:       Don't stop the browser's ongoing page-reading session.
   expectedActions: []
   produced:        ["browser.readPageContent"]

FAIL BAD_NEGATIVE sol   | target=browser.readPageContent
   utterance:       Do not read the contents of this page aloud.
   expectedActions: []
   produced:        ["browser.stopReadPageContent"]

FAIL BAD_NEGATIVE sol   | target=browser.closeWebPage
   utterance:       Don't close the current web page.
   expectedActions: []
   produced:        ["browser.openWebPage"]

FAIL BAD_NEGATIVE sol   | target=browser.closeAllWebPages
   utterance:       Leave all my shopping, banking, news, email, and calendar browser tabs alone; don't close any of them.
   expectedActions: []
   produced:        ["browser.stopReadPageContent"]

FAIL BAD_NEGATIVE sol   | target=browser.reloadPage
   utterance:       Please don't reload the page.
   expectedActions: []
   produced:        ["browser.reloadPage"]

FAIL BAD_NEGATIVE sol   | target=browser.captureScreenshot
   utterance:       Do not take a screenshot of the browser.
   expectedActions: []
   produced:        ["browser.captureScreenshot"]

FAIL BAD_NEGATIVE sol   | target=browser.zoomReset
   utterance:       Don't reset the browser zoom level.
   expectedActions: []
   produced:        ["browser.zoomReset"]

FAIL BAD_NEGATIVE sol   | target=browser.followLinkByText
   utterance:       Do not follow the 'Learn more' link on this page.
   expectedActions: []
   produced:        ["browser.stopReadPageContent"]

FAIL BAD_NEGATIVE sol   | target=browser.changeSearchProvider
   utterance:       Don't change my default search provider.
   expectedActions: []
   produced:        ["browser.changeSearchProvider"]

FAIL BAD_NEGATIVE sol   | target=browser.openWebPage
   utterance:       Do not open a new web page right now.
   expectedActions: []
   produced:        ["browser.closeWebPage"]

FAIL BAD_NEGATIVE sol   | target=desktop.AdjustVolume
   utterance:       Don't change the system volume.
   expectedActions: []
   produced:        ["desktop.AdjustVolume"]

FAIL BAD_NEGATIVE sol   | target=desktop.ApplyTheme
   utterance:       Do not apply a new desktop theme.
   expectedActions: []
   produced:        ["desktop.ApplyTheme"]

FAIL BAD_NEGATIVE sol   | target=desktop.CloseProgram
   utterance:       Please don't close any running programs.
   expectedActions: []
   produced:        ["desktop.CloseProgram"]

FAIL BAD_NEGATIVE sol   | target=list.clearList
   utterance:       Do not clear my shopping list.
   expectedActions: []
   produced:        ["list.removeItems"]

FAIL BAD_NEGATIVE sol   | target=timer.cancelReminder
   utterance:       Don't cancel my reminder.
   expectedActions: []
   produced:        ["timer.listReminders"]


=== summary (negative cases; PASS = no action produced) ===
sol    pass=1 fail=19 err=0 bad-negative-rate=95.0%

Command:

node src/translationBench/scripts/reproNegatives.mjs --api-key <key> --models sol --limit 20 --concurrency 6

Caveats

• The sol/terra/luna split is not yet valid. OPENAI_MODEL fixes the wire model, so --models currently selects an endpoint name rather than the served model; all arms hit the same one. Only single-model runs are meaningful until that is fixed. The run above is a single sol arm.
• Scoring is hand-rolled (produced.length !== expectedActions.length) rather than going through scoreTranslationBenchTranslationOutcome. A few of these 19 could score differently under real grading.
• Utterances are hand-written, not drawn from gold synthesizer data.

Draft — opening for visibility on the failure mode. Happy to wire the repo scorer and fix per-model routing before this is considered for merge.