Add pytorch to one CI configuration to increase test coverage

[mdboom]

None of our tests that depend on pytorch are currently covered in our CI. It's a heavy dependency to add /everywhere/ but it would be useful to add it to at least one configuration.

[mdboom]

Duplicate of #1987.