robots.txt is not valid

[MaxwellCohen]

Describe the bug

I ran a Pagespeed Insights Lighthouse test on a random npmx page, and it reported that the robots.txt is malformed. This allows bots not specifically banned to scan pages they should not be able to and verify it by seeing pages that should be disallowed are currently indexed by bots.

For example, a package code page like: https://npmx.dev/package-code/srilanka-api/v/1.0.6 is currently indexed by Google, and according to manual testing from robots.txt Validator and Testing Tool, it is allowed by non-ai crawlers like Google Bot even though it should be banned according to line

npmx.dev/public/robots.txt

Line 29 in 1ef8d3c

Disallow: /package-code/

The fix is adding the following code to the top of the robots.txt

User-agent: *

Additional context

No response

Logs