year 2038 problem in compileall.py

[bmwiedemann]

BPO	34990
Nosy	@jwilk, @serhiy-storchaka, @matrixise, @bmwiedemann, @ammaraskar, @miss-islington, @tirkarthi
PRs	WIP: bpo-34990: year 2038 problem in compileall.py #9892 bpo-34990: Change pyc headers to use 64-bit timestamps #19651 bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int #19708 [3.10] bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708) #27928 [3.9] bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708) #27929

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-08-24.15:13:34.226>
created_at = <Date 2018-10-15.11:22:16.253>
labels = ['build', '3.9', '3.10', '3.11']
title = 'year 2038 problem in compileall.py'
updated_at = <Date 2022-02-21.11:46:17.662>
user = 'https://github.com/bmwiedemann'

bugs.python.org fields:

activity = <Date 2022-02-21.11:46:17.662>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-08-24.15:13:34.226>
closer = 'ammar2'
components = ['Build']
creation = <Date 2018-10-15.11:22:16.253>
creator = 'bmwiedemann'
dependencies = []
files = []
hgrepos = []
issue_num = 34990
keywords = ['patch']
message_count = 13.0
messages = ['327743', '327747', '327748', '327749', '327750', '327751', '327753', '327774', '360270', '367679', '400200', '400214', '400215']
nosy_count = 7.0
nosy_names = ['jwilk', 'serhiy.storchaka', 'matrixise', 'bmwiedemann', 'ammar2', 'miss-islington', 'xtreak']
pr_nums = ['9892', '19651', '19708', '27928', '27929']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue34990'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[bmwiedemann]

To reproduce:
touch -d 2038-01-20 /usr/lib/python3.6/site-packages/six.py
python3 /usr/lib64/python3.6/compileall.py

File "/usr/lib64/python3.6/compileall.py", line 198, in compile_path
legacy=legacy, optimize=optimize)
File "/usr/lib64/python3.6/compileall.py", line 90, in compile_dir
legacy, optimize):
File "/usr/lib64/python3.6/compileall.py", line 138, in compile_file
mtime)
struct.error: 'l' format requires -2147483648 <= number <= 2147483647

It could use either
64 bit int (requires new .pyc format with different magic number) or
unsigned 32 bit int (gives us only another 68 years)

[matrixise]

With 3.8a

Traceback (most recent call last):
  File "/home/stephane/src/github.com/python/cpython/Lib/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/stephane/src/github.com/python/cpython/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/stephane/src/github.com/python/cpython/Lib/compileall.py", line 326, in <module>
    exit_status = int(not main())
  File "/home/stephane/src/github.com/python/cpython/Lib/compileall.py", line 303, in main
    if not compile_file(dest, args.ddir, args.force, args.rx,
  File "/home/stephane/src/github.com/python/cpython/Lib/compileall.py", line 142, in compile_file
    expect = struct.pack('<4sll', importlib.util.MAGIC_NUMBER,
struct.error: 'l' format requires -2147483648 <= number <= 2147483647

[matrixise]

But until 2038, maybe there will be a new format for the .pyc file.

We should keep this issue and try to fix it for 3.8 or 3.9?

[bmwiedemann]

It does not need to be fixed tomorrow, but 2037 is too late, because by then there will be a lot of legacy systems around.
(Un)fortunately many systems live 10+ years now

[vstinner]

Timestamp with year >= 2038 are accepted: importlib._bootstrap_external._code_to_timestamp_pyc() uses (int(x) & 0xFFFFFFFF). It's not a bug, but by design. compileall should just do the same. Sorry, I don't know if it's specified somewhere, but I know that it's done on purpose.

[matrixise]

So we need to fix compileall.py.

maybe we could add the label 'easy' to this issue.

[tirkarthi]

A reproducer in Python that can be added to test_compileall if needed :

def test_compile_all_2038(self):
    with open(self.source_path, 'r') as f:
        os.utime(f.name, (2147558400, 2147558400)) # Jan 20, 2038 as touch
    self.assertTrue(compileall.compile_file(pathlib.Path(self.source_path)))

./python.exe -m unittest -v test.test_compileall.CompileallTestsWithSourceEpoch.test_compile_all_2038
test_compile_all_2038 (test.test_compileall.CompileallTestsWithSourceEpoch) ... ERROR

======================================================================
ERROR: test_compile_all_2038 (test.test_compileall.CompileallTestsWithSourceEpoch)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/test_py_compile.py", line 30, in wrapper
    return fxn(*args, **kwargs)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/test_compileall.py", line 114, in test_compile_all_2038
    self.assertTrue(compileall.compile_file(pathlib.Path(self.source_path)))
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/compileall.py", line 142, in compile_file
    expect = struct.pack('<4sll', importlib.util.MAGIC_NUMBER,
struct.error: 'l' format requires -2147483648 <= number <= 2147483647

---

Thanks

[tirkarthi]

Victor seems there was some discussion about 2038 problem in the original PR but I don't know if it's related to this. Reference : #4575 (comment)

Thanks

[bmwiedemann]

ping.
Another 19th of January passed.

I'd still like to see progress on this, because this hinders my other y2038 bug discovery work.

[vstinner]

I would prefer to mimick importlib._bootstrap_external which uses:

def _pack_uint32(x):
    """Convert a 32-bit integer to little-endian."""
    return (int(x) & 0xFFFFFFFF).to_bytes(4, 'little')

Using 64-bit timestamp (PR 19651), treat timestamp as unsigned (PR 9892 and PR 19708) have drawback:

• 64-bit timestamp make .pyc files larger
• unsigned timestamp no longer support timestamp before 1969 which can cause practical issues

"& 0xFFFFFFFF" looks dead simple, uses a fixed size of 4 bytes and doesn't have any limitation of year 2038.

The timestamp doesn't have to be exact. In practice, it sounds very unlikely that two timestamps are equal when compared using (ts1 & 0xFFFFFFFF) == (ts2 & 0xFFFFFFFF). I expect file modification times to be close by a few days, not separated by 2**32 seconds (136 years).

Use hash based .pyc to avoid any issuse with file modification time: it should make Python more deterministic (more "reproducible").
https://docs.python.org/dev/reference/import.html#pyc-invalidation