Skip to content

fix: stop silently passing any script with 'inversion' in its path - #225

Merged
Jammy2211 merged 1 commit into
mainfrom
feature/howto-smoke-all-tutorials
Aug 4, 2026
Merged

fix: stop silently passing any script with 'inversion' in its path#225
Jammy2211 merged 1 commit into
mainfrom
feature/howto-smoke-all-tutorials

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Summary

execute_script() caught CalledProcessError and, if the string inversion appeared anywhere in the script path, recorded the run as PASSED with "Inversion script failure (ignored)". It matched on the filename, not the exception — so a script could fail for any reason at all and still be reported green.

Demonstrated on a file whose entire contents are raise SystemExit("deliberate failure"), named tutorial_3_inversions.py:

PyAutoHands console recorded status has_failures
main PASS (inversion, 0.0s) Status.PASSED False
this PR FAIL (0.0s) deliberate failure Status.FAILED True

Why it matters

Three scripts workspace-wide match the substring — and two are the chapter_4_pixelizations/tutorial_3_inversions.py files in HowToGalaxy and HowToLens:

HowToLens/scripts/chapter_4_pixelizations/tutorial_3_inversions.py
HowToGalaxy/scripts/chapter_4_pixelizations/tutorial_3_inversions.py
autolens_profiling/scripts/interferometer/likelihood_breakdown/datacube/inversion_setup_decompose.py

So the escape covered precisely the tutorial that shipped broken in PyAutoLabs/HowToGalaxy#56 / #57, and would have kept it green under any script run. execute_notebooks_in_folder never had this clause, which is exactly why the notebook job reported that failure while a script run never would have.

This blocks PyAutoLabs/HowToGalaxy#58, which expands HowTo smoke to run every script: without this change the newly-covered tutorial_3_inversions.py would be unconditionally green, defeating the point of the coverage work.

Deliberately left in place

execute_notebooks_in_folder has a narrower cousin — if "InversionException" in traceback.format_exc() → PASS. It keys off the exception type rather than the filename, so it is defensible for genuinely data-dependent inversion failures and is out of scope here. Flagging it for a separate decision. The adjacent is_clean_skip_exit() pass is correct as-is (a sys.exit(0) skip guard is a clean exit by design).

Test Plan

  • Negative control above: identical script, opposite recorded status before/after
  • Downstream suites run against this branch — HowToLens 39/39 green, HowToFit 15/15 green, HowToGalaxy 25/26 with the one genuine failure now correctly reported instead of silently passed
  • PyAutoHands test suite

Generated by the PyAutoLabs agent workflow.

execute_script() caught CalledProcessError and, if the string 'inversion'
appeared anywhere in the script path, recorded the run as PASSED with
'Inversion script failure (ignored)'. It was a substring match on the
path, not on the exception, so a script could fail for any reason at all
and still be reported green.

Demonstrated on a file containing only :

  before:  PASS (inversion)  status=PASSED  has_failures=False
  after:   FAIL              status=FAILED  has_failures=True

Three scripts workspace-wide match the substring, two of which are the
chapter_4 tutorial_3_inversions files in HowToGalaxy and HowToLens — so
the escape covered precisely the tutorial that shipped broken in
HowToGalaxy #56/#57 and would have kept it green under any script run.
The notebook runner never had this clause, which is why the notebook job
reported that failure while a script run would not have.

Its narrower cousin in execute_notebooks_in_folder (InversionException in
the traceback) is left in place: it keys off the exception type rather
than the filename, so it is deliberately out of scope here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant