From 3676169dc48d5962d8686b86aa871c657f80145a Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith [Google LLC]" Date: Fri, 9 Jun 2023 21:51:14 -0700 Subject: [PATCH 1/8] Cleanup and clarify our hashlib docs. --- Doc/library/hashlib.rst | 64 +++++++++++++++++++++++++++-------------- 1 file changed, 42 insertions(+), 22 deletions(-) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index 797870b9d7e260a..ae2190f62c8ba5d 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -11,7 +11,7 @@ .. index:: single: message digest, MD5 - single: secure hash algorithm, SHA1, SHA224, SHA256, SHA384, SHA512 + single: secure hash algorithm, SHA1, SHA2, SHA224, SHA256, SHA384, SHA512, SHA3, Shake, Blake2 .. testsetup:: @@ -64,18 +64,21 @@ concatenation of the data fed to it so far using the :meth:`digest` or .. index:: single: OpenSSL; (use in module hashlib) Constructors for hash algorithms that are always present in this module are -:func:`sha1`, :func:`sha224`, :func:`sha256`, :func:`sha384`, -:func:`sha512`, :func:`blake2b`, and :func:`blake2s`. -:func:`md5` is normally available as well, though it -may be missing or blocked if you are using a rare "FIPS compliant" build of Python. -Additional algorithms may also be available depending upon the OpenSSL -library that Python uses on your platform. On most platforms the +:func:`sha1`, :func:`sha224`, :func:`sha256`, :func:`sha384`, :func:`sha512`, :func:`sha3_224`, :func:`sha3_256`, :func:`sha3_384`, :func:`sha3_512`, -:func:`shake_128`, :func:`shake_256` are also available. +:func:`shake_128`, :func:`shake_256`, :func:`blake2b`, and :func:`blake2s`. +:func:`md5` is normally available as well, though it may be missing or blocked +if you are using a rare "FIPS compliant" build of Python. + +Additional algorithms may also be available depending upon the OpenSSL library +that Python uses on your platform but they are not guaranteed on all +installations and will only be accessible by name via :func:`new`, if present +at all. .. versionadded:: 3.6 SHA3 (Keccak) and SHAKE constructors :func:`sha3_224`, :func:`sha3_256`, - :func:`sha3_384`, :func:`sha3_512`, :func:`shake_128`, :func:`shake_256`. + :func:`sha3_384`, :func:`sha3_512`, :func:`shake_128`, :func:`shake_256` + were added. .. versionadded:: 3.6 :func:`blake2b` and :func:`blake2s` were added. @@ -89,7 +92,7 @@ library that Python uses on your platform. On most platforms the that the hashing algorithm is not used in a security context, e.g. as a non-cryptographic one-way compression function. - Hashlib now uses SHA3 and SHAKE from OpenSSL 1.1.1 and newer. + Hashlib now uses SHA3 and SHAKE from OpenSSL if it provides it. For example, to obtain the digest of the byte string ``b"Nobody inspects the spammish repetition"``:: @@ -108,20 +111,37 @@ More condensed: >>> hashlib.sha256(b"Nobody inspects the spammish repetition").hexdigest() '031edd7d41651593c5fe5c006fa5752b37fddff7bc4e843aa6af0c950f4b9406' -.. function:: new(name[, data], *, usedforsecurity=True) +.. function:: new(name[, data], \*, usedforsecurity=True) Is a generic constructor that takes the string *name* of the desired algorithm as its first parameter. It also exists to allow access to the above listed hashes as well as any other algorithms that your OpenSSL - library may offer. The named constructors are much faster than :func:`new` - and should be preferred. - -Using :func:`new` with an algorithm provided by OpenSSL: - - >>> h = hashlib.new('sha256') - >>> h.update(b"Nobody inspects the spammish repetition") + library may offer. + + Using :func:`new` with an algorithm name: + + >>> h = hashlib.new('sha256') + >>> h.update(b"Nobody inspects the spammish repetition") + >>> h.hexdigest() + '031edd7d41651593c5fe5c006fa5752b37fddff7bc4e843aa6af0c950f4b9406' + +These named constructors below are much faster than :func:`new` and should be +preferred: + +.. function:: md5([, data], \*, usedforsecurity=True) +.. function:: sha1([, data], \*, usedforsecurity=True) +.. function:: sha224([, data], \*, usedforsecurity=True) +.. function:: sha256([, data], \*, usedforsecurity=True) +.. function:: sha384([, data], \*, usedforsecurity=True) +.. function:: sha512([, data], \*, usedforsecurity=True) +.. function:: sha3_224([, data], \*, usedforsecurity=True) +.. function:: sha3_256([, data], \*, usedforsecurity=True) +.. function:: sha3_384([, data], \*, usedforsecurity=True) +.. function:: sha3_512([, data], \*, usedforsecurity=True) + + >>> h = hashlib.sha3_224(b"Nobody inspects the spammish repetition") >>> h.hexdigest() - '031edd7d41651593c5fe5c006fa5752b37fddff7bc4e843aa6af0c950f4b9406' + 'd6ff1bfb32cf835a3085ba18caa4a13f6307fdfb565521e0db35b064' Hashlib provides the following constant attributes: @@ -768,7 +788,7 @@ Domain Dedication 1.0 Universal: .. _BLAKE2: https://www.blake2.net .. _HMAC: https://en.wikipedia.org/wiki/Hash-based_message_authentication_code .. _BLAKE: https://web.archive.org/web/20200918190133/https://131002.net/blake/ -.. _SHA-3: https://en.wikipedia.org/wiki/NIST_hash_function_competition +.. _SHA-3: https://en.wikipedia.org/wiki/Secure_Hash_Algorithms .. _ChaCha: https://cr.yp.to/chacha.html .. _pyblake2: https://pythonhosted.org/pyblake2/ .. _NIST-SP-800-132: https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-132.pdf @@ -789,9 +809,9 @@ Domain Dedication 1.0 Universal: https://csrc.nist.gov/csrc/media/publications/fips/180/2/archive/2002-08-01/documents/fips180-2.pdf The FIPS 180-2 publication on Secure Hash Algorithms. - https://en.wikipedia.org/wiki/Cryptographic_hash_function#Cryptographic_hash_algorithms + https://en.wikipedia.org/wiki/Cryptographic_hash_function#Attacks_on_cryptographic_hash_algorithms Wikipedia article with information on which algorithms have known issues and - what that means regarding their use. + what that means regarding their use. (ex: MD5 and SHA1 are insecure) https://www.ietf.org/rfc/rfc8018.txt PKCS #5: Password-Based Cryptography Specification Version 2.1 From c4785129b6cba9d120da1285f12f11c1218ddbcf Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith [Google LLC]" Date: Fri, 9 Jun 2023 22:11:45 -0700 Subject: [PATCH 2/8] more updates --- Doc/library/hashlib.rst | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index ae2190f62c8ba5d..1c9f1804334f49e 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -34,7 +34,8 @@ digests. The modern term is secure hash. .. warning:: - Some algorithms have known hash collision weaknesses, refer to the "See + Some algorithms have known hash collision weaknesses (including MD5 and + SHA1). Refer to `Attacks on cryptographic hash alogorithms`_ and the "See also" section at the end. @@ -227,6 +228,9 @@ A hash object has the following methods: SHAKE variable length digests ----------------------------- +.. function:: shake_128([, data], \*, usedforsecurity=True) +.. function:: shake_256([, data], \*, usedforsecurity=True) + The :func:`shake_128` and :func:`shake_256` algorithms provide variable length digests with length_in_bits//2 up to 128 or 256 bits of security. As such, their digest methods require a length. Maximum length is not limited @@ -243,8 +247,13 @@ by the SHAKE algorithm. Like :meth:`digest` except the digest is returned as a string object of double length, containing only hexadecimal digits. This may be used to - exchange the value safely in email or other non-binary environments. + exchange the value in email or other non-binary environments. + +Example use: + >>> h = hashlib.shake_256(b'Nobody inspects the spammish repetition') + >>> h.hexdigest(20) + b'44709d6fcb83d92a76dcb0b668c98e1b1d3dafe7' File hashing ------------ @@ -793,6 +802,7 @@ Domain Dedication 1.0 Universal: .. _pyblake2: https://pythonhosted.org/pyblake2/ .. _NIST-SP-800-132: https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-132.pdf .. _stackexchange pbkdf2 iterations question: https://security.stackexchange.com/questions/3959/recommended-of-iterations-when-using-pbkdf2-sha256/ +.. _Attacks on cryptographic hash alogorithms: https://en.wikipedia.org/wiki/Cryptographic_hash_function#Attacks_on_cryptographic_hash_algorithms .. seealso:: @@ -803,15 +813,15 @@ Domain Dedication 1.0 Universal: Module :mod:`base64` Another way to encode binary hashes for non-binary environments. - https://www.blake2.net - Official BLAKE2 website. + https://nvlpubs.nist.gov/nistpubs/fips/nist.fips.180-4.pdf + The FIPS 180-4 publication on Secure Hash Algorithms. - https://csrc.nist.gov/csrc/media/publications/fips/180/2/archive/2002-08-01/documents/fips180-2.pdf - The FIPS 180-2 publication on Secure Hash Algorithms. + https://www.blake2.net/ + Official BLAKE2 website. https://en.wikipedia.org/wiki/Cryptographic_hash_function#Attacks_on_cryptographic_hash_algorithms Wikipedia article with information on which algorithms have known issues and - what that means regarding their use. (ex: MD5 and SHA1 are insecure) + what that means regarding their use. https://www.ietf.org/rfc/rfc8018.txt PKCS #5: Password-Based Cryptography Specification Version 2.1 From b76b6d52d2ebb537a43f418f6fcdc4a0a6ad7187 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith [Google LLC]" Date: Fri, 9 Jun 2023 22:19:45 -0700 Subject: [PATCH 3/8] more more --- Doc/library/hashlib.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index 1c9f1804334f49e..20ac121f903e009 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -22,7 +22,8 @@ This module implements a common interface to many different secure hash and message digest algorithms. Included are the FIPS secure hash algorithms SHA1, -SHA224, SHA256, SHA384, and SHA512 (defined in FIPS 180-2) as well as RSA's MD5 +SHA224, SHA256, SHA384, SHA512, (defined in `the FIPS 180-4 standard`_), +the SHA-3 series (defined in `the FIPS 202 standard`_) as well as RSA's MD5 algorithm (defined in internet :rfc:`1321`). The terms "secure hash" and "message digest" are interchangeable. Older algorithms were called message digests. The modern term is secure hash. @@ -803,6 +804,8 @@ Domain Dedication 1.0 Universal: .. _NIST-SP-800-132: https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-132.pdf .. _stackexchange pbkdf2 iterations question: https://security.stackexchange.com/questions/3959/recommended-of-iterations-when-using-pbkdf2-sha256/ .. _Attacks on cryptographic hash alogorithms: https://en.wikipedia.org/wiki/Cryptographic_hash_function#Attacks_on_cryptographic_hash_algorithms +.. _the FIPS 180-4 standard: https://csrc.nist.gov/publications/detail/fips/180/4/final +.. _the FIPS 202 standard: https://csrc.nist.gov/publications/detail/fips/202/final .. seealso:: @@ -816,6 +819,9 @@ Domain Dedication 1.0 Universal: https://nvlpubs.nist.gov/nistpubs/fips/nist.fips.180-4.pdf The FIPS 180-4 publication on Secure Hash Algorithms. + https://csrc.nist.gov/publications/detail/fips/202/final + The FIPS 202 publication on the SHA-3 Standard. + https://www.blake2.net/ Official BLAKE2 website. From 061885253b84f47eb17ce2ac73cb776ddf477264 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith [Google LLC]" Date: Fri, 9 Jun 2023 23:42:10 -0700 Subject: [PATCH 4/8] minor tweaks --- Doc/library/hashlib.rst | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index 20ac121f903e009..52233628f10ead0 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -120,15 +120,13 @@ More condensed: above listed hashes as well as any other algorithms that your OpenSSL library may offer. - Using :func:`new` with an algorithm name: +Using :func:`new` with an algorithm name: - >>> h = hashlib.new('sha256') - >>> h.update(b"Nobody inspects the spammish repetition") - >>> h.hexdigest() - '031edd7d41651593c5fe5c006fa5752b37fddff7bc4e843aa6af0c950f4b9406' + >>> h = hashlib.new('sha256') + >>> h.update(b"Nobody inspects the spammish repetition") + >>> h.hexdigest() + '031edd7d41651593c5fe5c006fa5752b37fddff7bc4e843aa6af0c950f4b9406' -These named constructors below are much faster than :func:`new` and should be -preferred: .. function:: md5([, data], \*, usedforsecurity=True) .. function:: sha1([, data], \*, usedforsecurity=True) @@ -141,11 +139,10 @@ preferred: .. function:: sha3_384([, data], \*, usedforsecurity=True) .. function:: sha3_512([, data], \*, usedforsecurity=True) - >>> h = hashlib.sha3_224(b"Nobody inspects the spammish repetition") - >>> h.hexdigest() - 'd6ff1bfb32cf835a3085ba18caa4a13f6307fdfb565521e0db35b064' +Named constructors such as these are faster than passing an algorithm name to +:func:`new`. -Hashlib provides the following constant attributes: +Hashlib provides the following constant module attributes: .. data:: algorithms_guaranteed From 4f53fa973231a098669ae227990f0317686c055d Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith [Google LLC]" Date: Fri, 9 Jun 2023 23:47:16 -0700 Subject: [PATCH 5/8] more... --- Doc/library/hashlib.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index 52233628f10ead0..85ca8a1f38bbffe 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -71,11 +71,13 @@ Constructors for hash algorithms that are always present in this module are :func:`shake_128`, :func:`shake_256`, :func:`blake2b`, and :func:`blake2s`. :func:`md5` is normally available as well, though it may be missing or blocked if you are using a rare "FIPS compliant" build of Python. +These correspond to :data:`algorithms_guaranteed`. -Additional algorithms may also be available depending upon the OpenSSL library -that Python uses on your platform but they are not guaranteed on all -installations and will only be accessible by name via :func:`new`, if present -at all. +Additional algorithms may also be available depending upon the specific OpenSSL +library that :mod:`hashlib` may have been linked against on your platform. They +they *are not guaranteed available* on all installations and, when present, +will only be accessible by name via :func:`new`. See +:data:`algorithms_available`. .. versionadded:: 3.6 SHA3 (Keccak) and SHAKE constructors :func:`sha3_224`, :func:`sha3_256`, From 7099108c614178e3248682477c2b7f5db0c4e22a Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith [Google LLC]" Date: Sat, 10 Jun 2023 11:04:54 -0700 Subject: [PATCH 6/8] rearranged, improved text and crosslinks. --- Doc/library/hashlib.rst | 62 ++++++++++++++++++++++------------------- 1 file changed, 34 insertions(+), 28 deletions(-) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index 85ca8a1f38bbffe..870129de212768c 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -33,12 +33,6 @@ digests. The modern term is secure hash. If you want the adler32 or crc32 hash functions, they are available in the :mod:`zlib` module. -.. warning:: - - Some algorithms have known hash collision weaknesses (including MD5 and - SHA1). Refer to `Attacks on cryptographic hash alogorithms`_ and the "See - also" section at the end. - .. _hash-algorithms: @@ -46,22 +40,17 @@ Hash algorithms --------------- There is one constructor method named for each type of :dfn:`hash`. All return -a hash object with the same simple interface. For example: use :func:`sha256` to -create a SHA-256 hash object. You can now feed this object with :term:`bytes-like -objects ` (normally :class:`bytes`) using the :meth:`update` method. -At any point you can ask it for the :dfn:`digest` of the -concatenation of the data fed to it so far using the :meth:`digest` or -:meth:`hexdigest` methods. +a hash object with the same simple interface. For example: use :func:`sha256` +to create a SHA-256 hash object. You can now feed this object with +:term:`bytes-like objects ` (normally :class:`bytes`) using +the :meth:`update` method. At any point you can ask it for the +:dfn:`digest` of the concatenation of the data fed to it so far using the +:meth:`digest()` or :meth:`hexdigest()` methods. -.. note:: - - For better multithreading performance, the Python :term:`GIL` is released for - data larger than 2047 bytes at object creation or on update. - -.. note:: +To allow multithreading, the Python :term:`GIL` is released while computing a +hash supplied more than 2047 bytes of data at once in its constructor or +:meth:`.update` method. - Feeding string objects into :meth:`update` is not supported, as hashes work - on bytes, not on characters. .. index:: single: OpenSSL; (use in module hashlib) @@ -73,11 +62,16 @@ Constructors for hash algorithms that are always present in this module are if you are using a rare "FIPS compliant" build of Python. These correspond to :data:`algorithms_guaranteed`. -Additional algorithms may also be available depending upon the specific OpenSSL -library that :mod:`hashlib` may have been linked against on your platform. They -they *are not guaranteed available* on all installations and, when present, -will only be accessible by name via :func:`new`. See -:data:`algorithms_available`. +Additional algorithms may also be available if your Python distribution's +:mod:`hashlib` was linked against a build of OpenSSL that provides others. +Others *are not guaranteed available* on all installations and will only be +accessible by name via :func:`new`. See :data:`algorithms_available`. + +.. warning:: + + Some algorithms have known hash collision weaknesses (including MD5 and + SHA1). Refer to `Attacks on cryptographic hash alogorithms`_ and the "See + also" section at the end. .. versionadded:: 3.6 SHA3 (Keccak) and SHAKE constructors :func:`sha3_224`, :func:`sha3_256`, @@ -96,10 +90,14 @@ will only be accessible by name via :func:`new`. See that the hashing algorithm is not used in a security context, e.g. as a non-cryptographic one-way compression function. +.. versionchanged:: 3.9 Hashlib now uses SHA3 and SHAKE from OpenSSL if it provides it. -For example, to obtain the digest of the byte string ``b"Nobody inspects the -spammish repetition"``:: +Usage +----- + +To obtain the digest of the byte string ``b"Nobody inspects the spammish +repetition"``:: >>> import hashlib >>> m = hashlib.sha256() @@ -115,6 +113,9 @@ More condensed: >>> hashlib.sha256(b"Nobody inspects the spammish repetition").hexdigest() '031edd7d41651593c5fe5c006fa5752b37fddff7bc4e843aa6af0c950f4b9406' +Constructors +------------ + .. function:: new(name[, data], \*, usedforsecurity=True) Is a generic constructor that takes the string *name* of the desired @@ -144,6 +145,9 @@ Using :func:`new` with an algorithm name: Named constructors such as these are faster than passing an algorithm name to :func:`new`. +Attributes +---------- + Hashlib provides the following constant module attributes: .. data:: algorithms_guaranteed @@ -165,10 +169,12 @@ Hashlib provides the following constant module attributes: .. versionadded:: 3.2 +Hash Objects +------------ + The following values are provided as constant attributes of the hash objects returned by the constructors: - .. data:: hash.digest_size The size of the resulting hash in bytes. From 3f1898c55664da5d07fd30f6a1dd154190f2b648 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith [Google LLC]" Date: Sat, 10 Jun 2023 11:27:54 -0700 Subject: [PATCH 7/8] tyops --- Doc/library/hashlib.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index 870129de212768c..8f5c7ebea56f2d5 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -70,8 +70,8 @@ accessible by name via :func:`new`. See :data:`algorithms_available`. .. warning:: Some algorithms have known hash collision weaknesses (including MD5 and - SHA1). Refer to `Attacks on cryptographic hash alogorithms`_ and the "See - also" section at the end. + SHA1). Refer to `Attacks on cryptographic hash algorithms`_ and the + `hashlib-seealso`_ section at the end of this document. .. versionadded:: 3.6 SHA3 (Keccak) and SHAKE constructors :func:`sha3_224`, :func:`sha3_256`, @@ -808,11 +808,13 @@ Domain Dedication 1.0 Universal: .. _pyblake2: https://pythonhosted.org/pyblake2/ .. _NIST-SP-800-132: https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-132.pdf .. _stackexchange pbkdf2 iterations question: https://security.stackexchange.com/questions/3959/recommended-of-iterations-when-using-pbkdf2-sha256/ -.. _Attacks on cryptographic hash alogorithms: https://en.wikipedia.org/wiki/Cryptographic_hash_function#Attacks_on_cryptographic_hash_algorithms +.. _Attacks on cryptographic hash algorithms: https://en.wikipedia.org/wiki/Cryptographic_hash_function#Attacks_on_cryptographic_hash_algorithms .. _the FIPS 180-4 standard: https://csrc.nist.gov/publications/detail/fips/180/4/final .. _the FIPS 202 standard: https://csrc.nist.gov/publications/detail/fips/202/final +.. _hashlib-seealso: + .. seealso:: Module :mod:`hmac` @@ -830,9 +832,9 @@ Domain Dedication 1.0 Universal: https://www.blake2.net/ Official BLAKE2 website. - https://en.wikipedia.org/wiki/Cryptographic_hash_function#Attacks_on_cryptographic_hash_algorithms - Wikipedia article with information on which algorithms have known issues and - what that means regarding their use. + https://en.wikipedia.org/wiki/Cryptographic_hash_function + Wikipedia article with information on which algorithms have known issues + and what that means regarding their use. https://www.ietf.org/rfc/rfc8018.txt PKCS #5: Password-Based Cryptography Specification Version 2.1 From 96e57c8bfbf4386681c15dcd6fdd83905c613b54 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith [Google LLC]" Date: Sat, 10 Jun 2023 11:41:14 -0700 Subject: [PATCH 8/8] doctest pedantry FTW --- Doc/library/hashlib.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index 8f5c7ebea56f2d5..7f06151675c7f02 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -259,7 +259,7 @@ Example use: >>> h = hashlib.shake_256(b'Nobody inspects the spammish repetition') >>> h.hexdigest(20) - b'44709d6fcb83d92a76dcb0b668c98e1b1d3dafe7' + '44709d6fcb83d92a76dcb0b668c98e1b1d3dafe7' File hashing ------------