diff --git a/Doc/c-api/lifecycle.dot.pdf b/Doc/c-api/lifecycle.dot.pdf deleted file mode 100644 index ed5b5039c83e2c4..000000000000000 Binary files a/Doc/c-api/lifecycle.dot.pdf and /dev/null differ diff --git a/Doc/c-api/lifecycle.rst b/Doc/c-api/lifecycle.rst index 531c4080a0131c3..5e3dca29f2ad901 100644 --- a/Doc/c-api/lifecycle.rst +++ b/Doc/c-api/lifecycle.rst @@ -19,7 +19,7 @@ object's life. An arrow from *A* to *B* indicates that event *B* can occur after event *A* has occurred, with the arrow's label indicating the condition that must be true for *B* to occur after *A*. -.. only:: html and not epub +.. only:: builder_html .. raw:: html @@ -50,20 +50,13 @@ that must be true for *B* to occur after *A*. })(); -.. only:: epub or not (html or latex) +.. only:: not builder_html .. image:: lifecycle.dot.svg :align: center :class: invert-in-dark-mode :alt: Diagram showing events in an object's life. Explained in detail below. -.. only:: latex - - .. image:: lifecycle.dot.pdf - :align: center - :class: invert-in-dark-mode - :alt: Diagram showing events in an object's life. Explained in detail below. - .. container:: :name: life-events-graph-description diff --git a/Doc/library/http.server.rst b/Doc/library/http.server.rst index c4b9173f9e34eb2..0d427db19c0fcdb 100644 --- a/Doc/library/http.server.rst +++ b/Doc/library/http.server.rst @@ -39,7 +39,17 @@ handler. Code to create and run the server looks like this:: This class builds on the :class:`~socketserver.TCPServer` class by storing the server address as instance variables named :attr:`server_name` and :attr:`server_port`. The server is accessible by the handler, typically - through the handler's :attr:`server` instance variable. + through the handler's :attr:`~socketserver.BaseRequestHandler.server` + instance variable. + + .. attribute:: server_name + + The HTTP server's fully qualified domain name. + + .. attribute:: server_port + + The HTTP server's port number obtained from *server_address*. + .. class:: ThreadingHTTPServer(server_address, RequestHandlerClass) @@ -60,7 +70,7 @@ handler. Code to create and run the server looks like this:: object fails with a :exc:`RuntimeError`. The *certfile* argument is the path to the SSL certificate chain file, - and the *keyfile* is the path to file containing the private key. + and the *keyfile* is the path to the file containing the private key. A *password* can be specified for files protected and wrapped with PKCS#8, but beware that this could possibly expose hardcoded passwords in clear. @@ -140,7 +150,7 @@ instantiation, of which this module provides three different variants: .. attribute:: path - Contains the request path. If query component of the URL is present, + Contains the request path. If the query component of the URL is present, then ``path`` includes the query. Using the terminology of :rfc:`3986`, ``path`` here includes ``hier-part`` and the ``query``. @@ -190,7 +200,7 @@ instantiation, of which this module provides three different variants: Specifies a format string that should be used by :meth:`send_error` method for building an error response to the client. The string is filled by default with variables from :attr:`responses` based on the status code - that passed to :meth:`send_error`. + passed to :meth:`send_error`. .. attribute:: error_content_type @@ -238,8 +248,8 @@ instantiation, of which this module provides three different variants: .. method:: handle_expect_100() When an HTTP/1.1 conformant server receives an ``Expect: 100-continue`` - request header it responds back with a ``100 Continue`` followed by ``200 - OK`` headers. + request header it responds with a ``100 Continue`` followed by ``200 OK`` + headers. This method can be overridden to raise an error if the server does not want the client to continue. For example, the server can choose to send ``417 Expectation Failed`` as a response header and ``return False``. @@ -295,8 +305,8 @@ instantiation, of which this module provides three different variants: .. method:: send_response_only(code, message=None) Sends the response header only, used for the purposes when ``100 - Continue`` response is sent by the server to the client. The headers not - buffered and sent directly the output stream.If the *message* is not + Continue`` response is sent by the server to the client. The headers are + not buffered and sent directly the output stream. If the *message* is not specified, the HTTP message corresponding the response *code* is sent. This method does not reject *message* containing CRLF sequences. @@ -338,7 +348,7 @@ instantiation, of which this module provides three different variants: to create custom error logging mechanisms. The *format* argument is a standard printf-style format string, where the additional arguments to :meth:`log_message` are applied as inputs to the formatting. The client - ip address and current date and time are prefixed to every message logged. + IP address and current date and time are prefixed to every message logged. .. method:: version_string() @@ -402,6 +412,14 @@ instantiation, of which this module provides three different variants: .. versionadded:: 3.15 + .. attribute:: index_pages + + Specifies the filenames that are treated as directory index pages. + + Defaults to ``("index.html", "index.htm")``. + + .. versionadded:: 3.12 + .. attribute:: extensions_map A dictionary mapping suffixes into MIME types, contains custom overrides @@ -434,8 +452,8 @@ instantiation, of which this module provides three different variants: The request is mapped to a local file by interpreting the request as a path relative to the current working directory. - If the request was mapped to a directory, the directory is checked for a - file named ``index.html`` or ``index.htm`` (in that order). If found, the + If the request was mapped to a directory, the directory is checked for + an index page as specified by :attr:`index_pages`. If found, the file's contents are returned; otherwise a directory listing is generated by calling the :meth:`list_directory` method. This method uses :func:`os.listdir` to scan the directory, and returns a ``404`` error @@ -465,9 +483,32 @@ instantiation, of which this module provides three different variants: .. versionchanged:: 3.7 Support of the ``'If-Modified-Since'`` header. -The :class:`SimpleHTTPRequestHandler` class can be used in the following -manner in order to create a very basic webserver serving files relative to -the current directory:: + .. method:: list_directory(path) + + Helper to list the contents of *path* when no index page is present. + + This returns either a :term:`file-like object` (which must be closed + by the caller) or ``None`` to indicate an error, in which case the + caller has nothing further to do. In either case, the headers are sent. + + .. method:: guess_type(path) + + Guess the type of the file at the given *path*. + + This returns a string of the form ``type/subtype``, usable for + a MIME Content-type header. + + The default implementation looks the file's extension up in + :attr:`extensions_map`, falling back to + :func:`mimetypes.guess_file_type` and then to + :attr:`default_content_type`. + + .. versionchanged:: 3.13 + Add :func:`mimetypes.guess_file_type` as a fallback. + + +The :class:`SimpleHTTPRequestHandler` class can be used to create a very basic +webserver serving files relative to the current directory as follows:: import http.server import socketserver @@ -483,7 +524,7 @@ the current directory:: :class:`SimpleHTTPRequestHandler` can also be subclassed to enhance behavior, such as using different index file names by overriding the class attribute -:attr:`index_pages`. +:attr:`~SimpleHTTPRequestHandler.index_pages`. .. _http-server-cli: @@ -599,8 +640,8 @@ The following options are accepted: .. option:: -H, --header
- Specify an additional extra HTTP Response Header to send on successful HTTP - 200 responses. Can be used multiple times to send additional custom response + Specify an additional HTTP Response Header to send on successful HTTP 200 + responses. Can be used multiple times to send additional custom response headers. Headers that are sent automatically by the server (for instance Content-Type) will not be overwritten by the server. @@ -615,13 +656,13 @@ Security considerations .. index:: pair: http.server; security :class:`SimpleHTTPRequestHandler` will follow symbolic links when handling -requests, this makes it possible for files outside of the specified directory +requests which makes it possible for files outside of the specified directory to be served. Methods :meth:`BaseHTTPRequestHandler.send_header` and :meth:`BaseHTTPRequestHandler.send_response_only` assume sanitized input and do not perform input validation such as checking for the presence of CRLF -sequences. Untrusted input may result in HTTP Header injection attacks. +sequences. Untrusted input may result in HTTP header injection attacks. Earlier versions of Python did not scrub control characters from the log messages emitted to stderr from ``python -m http.server`` or the diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index db17f6b79a7c50d..dbf4560ab5e53b3 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -186,6 +186,9 @@ enclosed with either parentheses or double quotes) each string is quoted. However, the *password* argument to the ``LOGIN`` command is always quoted. If you want to avoid having an argument string quoted (eg: the *flags* argument to ``STORE``) then enclose the string in parentheses (eg: ``r'(\Deleted)'``). +In general, pass arguments unquoted and let the module quote them as needed. +An argument that is already enclosed in double quotes is left unchanged, +so that code which quotes arguments itself keeps working. Most commands return a tuple: ``(type, [data, ...])`` where *type* is usually ``'OK'`` or ``'NO'``, and *data* is either the text from the command response, @@ -410,7 +413,7 @@ An :class:`IMAP4` instance has the following methods: .. versionadded:: 3.14 -.. method:: IMAP4.list([directory[, pattern]]) +.. method:: IMAP4.list(directory='', pattern='*') List mailbox names in *directory* matching *pattern*. *directory* defaults to the top-level mail folder, and *pattern* defaults to match anything. Returned @@ -440,7 +443,7 @@ An :class:`IMAP4` instance has the following methods: The method no longer ignores silently arbitrary exceptions. -.. method:: IMAP4.lsub(directory='""', pattern='*') +.. method:: IMAP4.lsub(directory='', pattern='*') List subscribed mailbox names in directory matching pattern. *directory* defaults to the top level directory and *pattern* defaults to match any mailbox. diff --git a/Doc/library/re.rst b/Doc/library/re.rst index 617dc96f479926c..1aefb91e0ceb405 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -531,8 +531,9 @@ The special characters are: *name* exists, and with ``no-pattern`` if it doesn't. ``no-pattern`` is optional and can be omitted. For example, ``(<)?(\w+@\w+(?:\.\w+)+)(?(1)>|$)`` is a poor email matching pattern, which - will match with ``''`` as well as ``'user@host.com'``, but - not with ``''``. + matches ``''`` as well as ``'user@host.com'``, but does not + match ``''`` in their entirety + (:func:`re.search` finds only ``'user@host.com'`` in the former). .. versionchanged:: 3.12 Group *id* can only contain ASCII digits. diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst index 8c4678b4b23c778..ca034fcdc09b439 100644 --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -656,6 +656,9 @@ method on it, and to change its value you call the :meth:`!set` method. If you follow this protocol, the widget will always track the value of the variable, with no further intervention on your part. +Keep a reference to the variable for as long as a widget uses it, for example +by storing it as an attribute (see :class:`Variable`). + For example:: import tkinter as tk @@ -734,6 +737,8 @@ Here are some examples of typical usage:: myapp.mainloop() +.. _Tk-option-data-types: + Tk option data types ^^^^^^^^^^^^^^^^^^^^ @@ -767,12 +772,16 @@ color represent any legal hex digit. See page 160 of Ousterhout's book for details. cursor - The standard X cursor names from :file:`cursorfont.h` can be used, without the - ``XC_`` prefix. For example to get a hand cursor (``XC_hand2``), use the - string ``"hand2"``. You can also specify a bitmap and mask file of your own. + The name of the mouse cursor to display while the pointer is over the widget. + Tk provides a portable set of cursor names available on all platforms + (for example ``"arrow"``, ``"watch"``, ``"cross"``, or ``"hand2"``); + the standard X cursor names from :file:`cursorfont.h` may also be used, + without the ``XC_`` prefix (so ``XC_hand2`` becomes ``"hand2"``). + The full list of names, including the platform-specific ones, + is given in the :manpage:`cursors(3tk)` manual page. + You can also specify a bitmap and mask file of your own. On Windows a cursor file (:file:`.cur` or :file:`.ani`) may be used directly, giving its path preceded with an ``@``, as in ``"@C:/cursors/bart.ani"``. - See page 179 of Ousterhout's book. distance Screen distances can be specified in either pixels or absolute distances. @@ -2071,6 +2080,7 @@ Base and mixin classes Return the geometry of the widget, in the form ``widthxheight+x+y``. All dimensions are in pixels. + An offset can be negative; see :meth:`~Wm.geometry`. .. method:: winfo_height() @@ -2541,6 +2551,8 @@ Base and mixin classes *width* and *height* are in pixels (or grid units for a gridded window); a position preceded by ``+`` is measured from the left or top edge of the screen and one preceded by ``-`` from the right or bottom edge. + An offset can be negative, as in ``'200x100+-9+-8'``, when the window + edge is positioned beyond the corresponding screen edge. An empty string cancels any user-specified geometry, letting the window revert to its natural size. With no argument, return the current geometry as a string of the form @@ -5916,6 +5928,14 @@ Variable classes :class:`StringVar`, :class:`IntVar`, :class:`DoubleVar` or :class:`BooleanVar` -- rather than :class:`!Variable` directly. + .. note:: + + When a :class:`!Variable` is garbage collected, its Tcl variable is unset. + Keep a reference to it for as long as a widget is linked to it, for example + by storing it as an attribute rather than in a local variable. + Otherwise Tk recreates the Tcl variable to keep the widget working, but it + is never unset again, leaking one Tcl variable per dropped wrapper. + .. versionchanged:: 3.10 Two variables now compare equal (``==``) only when they have the same name, are of the same class, and belong to the same Tcl interpreter. diff --git a/Doc/library/tkinter.ttk.rst b/Doc/library/tkinter.ttk.rst index ef7bbd130fb8883..13b374e23558fc3 100644 --- a/Doc/library/tkinter.ttk.rst +++ b/Doc/library/tkinter.ttk.rst @@ -116,9 +116,10 @@ All the :mod:`!ttk` Widgets accept the following options: | | read-only, and may only be specified when the window is | | | created. | +-----------+--------------------------------------------------------------+ -| cursor | Specifies the mouse cursor to be used for the widget. If set | -| | to the empty string (the default), the cursor is inherited | -| | from the parent widget. | +| cursor | Specifies the mouse cursor to be used for the widget. See | +| | the *cursor* option type under :ref:`Tk-option-data-types`. | +| | If set to the empty string (the default), the cursor is | +| | inherited from the parent widget. | +-----------+--------------------------------------------------------------+ | takefocus | Determines whether the window accepts the focus during | | | keyboard traversal. 0, 1 or an empty string is returned. | diff --git a/Doc/library/turtle-star.pdf b/Doc/library/turtle-star.pdf deleted file mode 100644 index e354073dd42f5e6..000000000000000 Binary files a/Doc/library/turtle-star.pdf and /dev/null differ diff --git a/Doc/library/turtle-star.ps b/Doc/library/turtle-star.ps deleted file mode 100644 index 46362cb9f7c8f52..000000000000000 Binary files a/Doc/library/turtle-star.ps and /dev/null differ diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index 20c659756fe1c19..d7839e2f81172c5 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -46,7 +46,7 @@ direction it is facing, drawing a line as it moves. Give it the command Turtle can draw intricate shapes using programs that repeat simple moves. - .. image:: turtle-star.* + .. image:: turtle-star.png :align: center In Python, turtle graphics provides a representation of a physical "turtle" diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index b3168bb85b4765d..e74f3262ed540cd 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -279,12 +279,12 @@ and the exception occurs in the :keyword:`!try` clause of the inner handler, the outer handler will not handle the exception.) When an exception has been assigned using ``as target``, it is cleared at the -end of the :keyword:`!except` clause. This is as if :: +end of the :keyword:`!except` clause. This is as if:: except E as N: foo -was translated to :: +was translated to:: except E as N: try: @@ -341,7 +341,7 @@ can have either :keyword:`except` or :keyword:`!except*` clauses, but not both. The exception type for matching is mandatory in the case of :keyword:`!except*`, so ``except*:`` is a syntax error. The type is interpreted as in the case of :keyword:`!except`, but matching is performed on the exceptions contained in the -group that is being handled. An :exc:`TypeError` is raised if a matching +group that is being handled. A :exc:`TypeError` is raised if a matching type is a subclass of :exc:`!BaseExceptionGroup`, because that would have ambiguous semantics. @@ -357,7 +357,7 @@ or the last :keyword:`!except*` clause has run. After all :keyword:`!except*` clauses execute, the group of unhandled exceptions is merged with any exceptions that were raised or re-raised from within -:keyword:`!except*` clauses. This merged exception group propagates on.:: +:keyword:`!except*` clauses. This merged exception group propagates on:: >>> try: ... raise ExceptionGroup("eg", @@ -1311,7 +1311,7 @@ mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default parameter value is in effect modified. This is generally not what was intended. A way around this is to use ``None`` as the default, and explicitly test for it in the body of the function, -e.g.:: +for example:: def whats_on_the_telly(penguin=None): if penguin is None: diff --git a/Doc/tools/.nitignore b/Doc/tools/.nitignore index 7b956f0ef0eb2c4..8ac0e8ffcffc277 100644 --- a/Doc/tools/.nitignore +++ b/Doc/tools/.nitignore @@ -9,7 +9,6 @@ Doc/library/ast.rst Doc/library/asyncio-extending.rst Doc/library/email.charset.rst Doc/library/email.parser.rst -Doc/library/http.server.rst Doc/library/importlib.rst Doc/library/logging.config.rst Doc/library/logging.handlers.rst diff --git a/Lib/http/server.py b/Lib/http/server.py index ebc85052aecb900..095b5744bd12fc6 100644 --- a/Lib/http/server.py +++ b/Lib/http/server.py @@ -334,10 +334,13 @@ def parse_request(self): raise ValueError("unreasonable length http version") version_number = int(version_number[0]), int(version_number[1]) except (ValueError, IndexError): + # Send the error response with a status line and headers. + self.request_version = '' self.send_error( HTTPStatus.BAD_REQUEST, "Bad request version (%r)" % version) return False + self.request_version = version if version_number >= (1, 1) and self.protocol_version >= "HTTP/1.1": self.close_connection = False if version_number >= (2, 0): @@ -345,9 +348,9 @@ def parse_request(self): HTTPStatus.HTTP_VERSION_NOT_SUPPORTED, "Invalid HTTP version (%s)" % base_version_number) return False - self.request_version = version if not 2 <= len(words) <= 3: + self.request_version = '' self.send_error( HTTPStatus.BAD_REQUEST, "Bad request syntax (%r)" % requestline) @@ -356,6 +359,7 @@ def parse_request(self): if len(words) == 2: self.close_connection = True if command != 'GET': + self.request_version = '' self.send_error( HTTPStatus.BAD_REQUEST, "Bad HTTP/0.9 request type (%r)" % command) diff --git a/Lib/imaplib.py b/Lib/imaplib.py index 70abfa741444c30..a871e509b0d82c6 100644 --- a/Lib/imaplib.py +++ b/Lib/imaplib.py @@ -130,6 +130,9 @@ _Literal = br'.*{(?P\d+)}$' _Untagged_status = br'\* (?P\d+) (?P[A-Z-]+)( (?P.*))?' _control_chars = re.compile(b'[\x00-\x1F\x7F]') +_non_astring_char = re.compile(br'[(){ \x00-\x1f\x7f-\xff%*\\"]') +_non_list_char = re.compile(br'[(){ \x00-\x1f\x7f-\xff\\"]') +_quoted = re.compile(br'"(?:[^"\\]|\\.)*+"') class IMAP4: @@ -503,8 +506,7 @@ def append(self, mailbox, flags, date_time, message, *, if not mailbox: mailbox = 'INBOX' if flags: - if (flags[0],flags[-1]) != ('(',')'): - flags = '(%s)' % flags + flags = self._set_quote(flags) else: flags = None if date_time: @@ -514,7 +516,7 @@ def append(self, mailbox, flags, date_time, message, *, if translate_line_endings: message = MapCRLF.sub(CRLF, message) self.literal = message - return self._simple_command(name, mailbox, flags, date_time) + return self._simple_command(name, self._astring(mailbox), flags, date_time) def authenticate(self, mechanism, authobject): @@ -539,7 +541,7 @@ def authenticate(self, mechanism, authobject): #if not cap in self.capabilities: # Let the server decide! # raise self.error("Server doesn't allow %s authentication." % mech) self.literal = _Authenticator(authobject).process - typ, dat = self._simple_command('AUTHENTICATE', mech) + typ, dat = self._simple_command('AUTHENTICATE', self._atom(mech)) if typ != 'OK': raise self.error(dat[-1].decode('utf-8', 'replace')) self.state = 'AUTH' @@ -584,7 +586,8 @@ def copy(self, message_set, new_mailbox): (typ, [data]) = .copy(message_set, new_mailbox) """ - return self._simple_command('COPY', message_set, new_mailbox) + return self._simple_command('COPY', self._sequence_set(message_set), + self._astring(new_mailbox)) def create(self, mailbox): @@ -592,7 +595,7 @@ def create(self, mailbox): (typ, [data]) = .create(mailbox) """ - return self._simple_command('CREATE', mailbox) + return self._simple_command('CREATE', self._astring(mailbox)) def delete(self, mailbox): @@ -600,14 +603,15 @@ def delete(self, mailbox): (typ, [data]) = .delete(mailbox) """ - return self._simple_command('DELETE', mailbox) + return self._simple_command('DELETE', self._astring(mailbox)) def deleteacl(self, mailbox, who): """Delete the ACLs (remove any rights) set for who on mailbox. (typ, [data]) = .deleteacl(mailbox, who) """ - return self._simple_command('DELETEACL', mailbox, who) + return self._simple_command('DELETEACL', self._astring(mailbox), + self._astring(who)) def enable(self, capability): """Send an RFC5161 enable string to the server. @@ -646,7 +650,8 @@ def fetch(self, message_set, message_parts): 'data' are tuples of message part envelope and data. """ name = 'FETCH' - typ, dat = self._simple_command(name, message_set, message_parts) + typ, dat = self._simple_command(name, self._sequence_set(message_set), + self._fetch_parts(message_parts)) return self._untagged_response(typ, dat, name) @@ -655,7 +660,7 @@ def getacl(self, mailbox): (typ, [data]) = .getacl(mailbox) """ - typ, dat = self._simple_command('GETACL', mailbox) + typ, dat = self._simple_command('GETACL', self._astring(mailbox)) return self._untagged_response(typ, dat, 'ACL') @@ -663,7 +668,8 @@ def getannotation(self, mailbox, entry, attribute): """(typ, [data]) = .getannotation(mailbox, entry, attribute) Retrieve ANNOTATIONs.""" - typ, dat = self._simple_command('GETANNOTATION', mailbox, entry, attribute) + typ, dat = self._simple_command('GETANNOTATION', self._astring(mailbox), + entry, attribute) return self._untagged_response(typ, dat, 'ANNOTATION') @@ -674,7 +680,7 @@ def getquota(self, root): (typ, [data]) = .getquota(root) """ - typ, dat = self._simple_command('GETQUOTA', root) + typ, dat = self._simple_command('GETQUOTA', self._astring(root)) return self._untagged_response(typ, dat, 'QUOTA') @@ -683,7 +689,7 @@ def getquotaroot(self, mailbox): (typ, [[QUOTAROOT responses...], [QUOTA responses]]) = .getquotaroot(mailbox) """ - typ, dat = self._simple_command('GETQUOTAROOT', mailbox) + typ, dat = self._simple_command('GETQUOTAROOT', self._astring(mailbox)) typ, quota = self._untagged_response(typ, dat, 'QUOTA') typ, quotaroot = self._untagged_response(typ, dat, 'QUOTAROOT') return typ, [quotaroot, quota] @@ -702,15 +708,16 @@ def idle(self, duration=None): return Idler(self, duration) - def list(self, directory='""', pattern='*'): + def list(self, directory='', pattern='*'): """List mailbox names in directory matching pattern. - (typ, [data]) = .list(directory='""', pattern='*') + (typ, [data]) = .list(directory='', pattern='*') 'data' is list of LIST responses. """ name = 'LIST' - typ, dat = self._simple_command(name, directory, pattern) + typ, dat = self._simple_command(name, self._astring(directory), + self._list_mailbox(pattern)) return self._untagged_response(typ, dat, name) @@ -721,7 +728,8 @@ def login(self, user, password): NB: 'password' will be quoted. """ - typ, dat = self._simple_command('LOGIN', user, self._quote(password)) + typ, dat = self._simple_command('LOGIN', self._astring(user), + self._quote(password)) if typ != 'OK': raise self.error(dat[-1].decode('UTF-8', 'replace')) self.state = 'AUTH' @@ -767,15 +775,16 @@ def logout(self): return typ, dat - def lsub(self, directory='""', pattern='*'): + def lsub(self, directory='', pattern='*'): """List 'subscribed' mailbox names in directory matching pattern. - (typ, [data, ...]) = .lsub(directory='""', pattern='*') + (typ, [data, ...]) = .lsub(directory='', pattern='*') 'data' are tuples of message part envelope and data. """ name = 'LSUB' - typ, dat = self._simple_command(name, directory, pattern) + typ, dat = self._simple_command(name, self._astring(directory), + self._list_mailbox(pattern)) return self._untagged_response(typ, dat, name) def myrights(self, mailbox): @@ -783,7 +792,7 @@ def myrights(self, mailbox): (typ, [data]) = .myrights(mailbox) """ - typ,dat = self._simple_command('MYRIGHTS', mailbox) + typ,dat = self._simple_command('MYRIGHTS', self._astring(mailbox)) return self._untagged_response(typ, dat, 'MYRIGHTS') def namespace(self): @@ -829,7 +838,7 @@ def proxyauth(self, user): """ name = 'PROXYAUTH' - return self._simple_command(name, user) + return self._simple_command(name, self._astring(user)) def rename(self, oldmailbox, newmailbox): @@ -837,7 +846,8 @@ def rename(self, oldmailbox, newmailbox): (typ, [data]) = .rename(oldmailbox, newmailbox) """ - return self._simple_command('RENAME', oldmailbox, newmailbox) + return self._simple_command('RENAME', self._astring(oldmailbox), + self._astring(newmailbox)) def search(self, charset, *criteria): @@ -849,10 +859,11 @@ def search(self, charset, *criteria): If UTF8 is enabled, charset MUST be None. """ name = 'SEARCH' - if charset: + if charset is not None: if self.utf8_enabled: raise IMAP4.error("Non-None charset not valid in UTF8 mode") - typ, dat = self._simple_command(name, 'CHARSET', charset, *criteria) + typ, dat = self._simple_command(name, + 'CHARSET', self._astring(charset), *criteria) else: typ, dat = self._simple_command(name, *criteria) return self._untagged_response(typ, dat, name) @@ -876,7 +887,7 @@ def select(self, mailbox='INBOX', readonly=False): name = 'EXAMINE' else: name = 'SELECT' - typ, dat = self._simple_command(name, mailbox) + typ, dat = self._simple_command(name, self._astring(mailbox)) if typ != 'OK': self.state = 'AUTH' # Might have been 'SELECTED' return typ, dat @@ -895,14 +906,15 @@ def setacl(self, mailbox, who, what): (typ, [data]) = .setacl(mailbox, who, what) """ - return self._simple_command('SETACL', mailbox, who, what) + return self._simple_command('SETACL', self._astring(mailbox), + self._astring(who), self._astring(what)) - def setannotation(self, *args): + def setannotation(self, mailbox, *args): """(typ, [data]) = .setannotation(mailbox[, entry, attribute]+) Set ANNOTATIONs.""" - typ, dat = self._simple_command('SETANNOTATION', *args) + typ, dat = self._simple_command('SETANNOTATION', self._astring(mailbox), *args) return self._untagged_response(typ, dat, 'ANNOTATION') @@ -911,7 +923,8 @@ def setquota(self, root, limits): (typ, [data]) = .setquota(root, limits) """ - typ, dat = self._simple_command('SETQUOTA', root, limits) + typ, dat = self._simple_command('SETQUOTA', self._astring(root), + self._set_quote(limits)) return self._untagged_response(typ, dat, 'QUOTA') @@ -923,8 +936,9 @@ def sort(self, sort_criteria, charset, *search_criteria): name = 'SORT' #if not name in self.capabilities: # Let the server decide! # raise self.error('unimplemented extension command: %s' % name) - if (sort_criteria[0],sort_criteria[-1]) != ('(',')'): - sort_criteria = '(%s)' % sort_criteria + sort_criteria = self._set_quote(sort_criteria) + if charset is not None: + charset = self._astring(charset) typ, dat = self._simple_command(name, sort_criteria, charset, *search_criteria) return self._untagged_response(typ, dat, name) @@ -961,7 +975,8 @@ def status(self, mailbox, names): name = 'STATUS' #if self.PROTOCOL_VERSION == 'IMAP4': # Let the server decide! # raise self.error('%s unimplemented in IMAP4 (obtain IMAP4rev1 server, or re-code)' % name) - typ, dat = self._simple_command(name, mailbox, names) + typ, dat = self._simple_command(name, self._astring(mailbox), + self._set_quote(names)) return self._untagged_response(typ, dat, name) @@ -970,9 +985,9 @@ def store(self, message_set, command, flags): (typ, [data]) = .store(message_set, command, flags) """ - if (flags[0],flags[-1]) != ('(',')'): - flags = '(%s)' % flags # Avoid quoting the flags - typ, dat = self._simple_command('STORE', message_set, command, flags) + flags = self._set_quote(flags) + typ, dat = self._simple_command('STORE', self._sequence_set(message_set), + command, flags) return self._untagged_response(typ, dat, 'FETCH') @@ -981,7 +996,7 @@ def subscribe(self, mailbox): (typ, [data]) = .subscribe(mailbox) """ - return self._simple_command('SUBSCRIBE', mailbox) + return self._simple_command('SUBSCRIBE', self._astring(mailbox)) def thread(self, threading_algorithm, charset, *search_criteria): @@ -990,7 +1005,10 @@ def thread(self, threading_algorithm, charset, *search_criteria): (type, [data]) = .thread(threading_algorithm, charset, search_criteria, ...) """ name = 'THREAD' - typ, dat = self._simple_command(name, threading_algorithm, charset, *search_criteria) + if charset is not None: + charset = self._astring(charset) + typ, dat = self._simple_command(name, self._atom(threading_algorithm), + charset, *search_criteria) return self._untagged_response(typ, dat, name) @@ -1011,7 +1029,31 @@ def uid(self, command, *args): (command, self.state, ', '.join(Commands[command]))) name = 'UID' - typ, dat = self._simple_command(name, command, *args) + if command == 'COPY': + message_set, new_mailbox = args + args = (self._sequence_set(message_set), + self._astring(new_mailbox)) + elif command == 'FETCH': + message_set, message_parts = args + args = (self._sequence_set(message_set), + self._fetch_parts(message_parts)) + elif command == 'STORE': + message_set, op, flags = args + args = (self._sequence_set(message_set), op, + self._set_quote(flags)) + elif command == 'SORT': + sort_criteria, charset, *search_criteria = args + if charset is not None: + charset = self._astring(charset) + args = (self._set_quote(sort_criteria), charset, + *search_criteria) + elif command == 'THREAD': + threading_algorithm, charset, *search_criteria = args + if charset is not None: + charset = self._astring(charset) + args = (self._atom(threading_algorithm), charset, + *search_criteria) + typ, dat = self._simple_command(name, self._atom(command), *args) if command in ('SEARCH', 'SORT', 'THREAD'): name = command else: @@ -1024,7 +1066,7 @@ def unsubscribe(self, mailbox): (typ, [data]) = .unsubscribe(mailbox) """ - return self._simple_command('UNSUBSCRIBE', mailbox) + return self._simple_command('UNSUBSCRIBE', self._astring(mailbox)) def unselect(self): @@ -1393,13 +1435,53 @@ def _new_tag(self): return tag - def _quote(self, arg): + def _atom(self, arg): + return arg + + def _sequence_set(self, arg): + return arg - arg = arg.replace('\\', '\\\\') - arg = arg.replace('"', '\\"') + def _set_quote(self, arg): + if arg and arg[0] == '(' and arg[-1] == ')': + return arg + return '(' + arg + ')' - return '"' + arg + '"' + def _fetch_parts(self, arg): + # "ALL", "FULL" and "FAST" are macros, not data item names; + # they cannot be enclosed in parentheses. + if arg.upper() in ('ALL', 'FULL', 'FAST'): + return arg + return self._set_quote(arg) + def _quote(self, arg): + if isinstance(arg, str): + arg = bytes(arg, self._encoding) + arg = arg.replace(b'\\', br'\\') + arg = arg.replace(b'"', br'\"') + return b'"' + arg + b'"' + + # For backward compatibility, an argument already enclosed in double + # quotes is left unquoted, so that code which quotes arguments itself + # keeps working. New code should pass arguments unquoted and let the + # module quote them as needed. + + def _astring(self, arg): + if isinstance(arg, str): + arg = bytes(arg, self._encoding) + if _quoted.fullmatch(arg): + return arg + if arg and _non_astring_char.search(arg) is None: + return arg + return self._quote(arg) + + def _list_mailbox(self, arg): + if isinstance(arg, str): + arg = bytes(arg, self._encoding) + if _quoted.fullmatch(arg): + return arg + if arg and _non_list_char.search(arg) is None: + return arg + return self._quote(arg) def _simple_command(self, name, *args): diff --git a/Lib/test/test_httpservers.py b/Lib/test/test_httpservers.py index d4ae032610a91e2..63f65a9c5cf47a9 100644 --- a/Lib/test/test_httpservers.py +++ b/Lib/test/test_httpservers.py @@ -386,6 +386,8 @@ def test_simple_get(self): def test_invalid_request(self): self.sock.send(b'POST /index.html\r\n') res = self.sock.recv(1024) + # The error response is not sent in the bare HTTP/0.9 style. + self.assertStartsWith(res, b'HTTP/1.0 400 ') self.assertIn(b"Bad HTTP/0.9 request type ('POST')", res) def test_single_request(self): @@ -1102,6 +1104,19 @@ def test_http_0_9(self): self.assertEqual(result[0], b'Data\r\n') self.verify_get_called() + @support.subTests('request,code', [ + (b'GET / FUBAR\r\n\r\n', 400), # bad version + (b'GET / HTTP/2.0\r\n\r\n', 505), # unsupported version + (b'GET\r\n', 400), # bad syntax + (b'POST /\r\n', 400), # bad HTTP/0.9 request type + ]) + def test_request_line_error_has_status_line(self, request, code): + self.handler = SocketlessRequestHandler() + result = self.send_typical_request(request) + self.assertStartsWith(result[0], b'HTTP/1.1 %d ' % code) + self.verify_expected_headers(result[1:result.index(b'\r\n')]) + self.assertFalse(self.handler.get_called) + def test_extra_space(self): result = self.send_typical_request( b'GET /spaced out HTTP/1.1\r\n' diff --git a/Lib/test/test_imaplib.py b/Lib/test/test_imaplib.py index 097056c91a7f89e..cdc2f9c8d852322 100644 --- a/Lib/test/test_imaplib.py +++ b/Lib/test/test_imaplib.py @@ -176,6 +176,65 @@ def test_imap4_host_default_value(self): imaplib.IMAP4() self.assertIn(cm.exception.errno, expected_errnos) + def test_astring(self): + m = imaplib.IMAP4.__new__(imaplib.IMAP4) + m._encoding = 'ascii' + # Plain atoms are left unquoted. + self.assertEqual(m._astring('INBOX'), b'INBOX') + self.assertEqual(m._astring(b'INBOX'), b'INBOX') + # Names with protocol-sensitive characters are quoted. + self.assertEqual(m._astring('New folder'), b'"New folder"') + self.assertEqual(m._astring('a"b'), b'"a\\"b"') + self.assertEqual(m._astring('a\\b'), b'"a\\\\b"') + self.assertEqual(m._astring(''), b'""') + self.assertEqual(m._astring('*'), b'"*"') + # A well-formed quoted string is passed through unchanged. + self.assertEqual(m._astring('"New folder"'), b'"New folder"') + self.assertEqual(m._astring('""'), b'""') + # Including a lenient (non-RFC) backslash escape, which the server + # may accept. + self.assertEqual(m._astring('"a\\b"'), b'"a\\b"') + # A string that only looks quoted but is not a single token is + # quoted as data, closing the argument injection vector. + self.assertEqual(m._astring('"a" SELECT evil "'), + b'"\\"a\\" SELECT evil \\""') + self.assertEqual(m._astring('"'), b'"\\""') + # Non-ASCII names are only allowed in a quoted string or a + # literal, never in an atom (RFC 6855). + m._encoding = 'utf-8' + self.assertEqual(m._astring('Entwürfe'), '"Entwürfe"'.encode()) + self.assertEqual(m._astring(b'Entw\xc3\xbcrfe'), b'"Entw\xc3\xbcrfe"') + + def test_astring_idempotent(self): + # Quoting an already quoted argument should not change it, so that + # quoting twice gives the same result as quoting once. + m = imaplib.IMAP4.__new__(imaplib.IMAP4) + m._encoding = 'ascii' + for arg in ['INBOX', 'New folder', 'a"b', 'a\\b', '', '*', '%', + '"New folder"', '""', '"a\\b"', '"a" SELECT evil "', + '"', 'a\tb', 'a\rb', '\x7f', '(a)', b'Entw\xc3\xbcrfe']: + with self.subTest(arg=arg): + once = m._astring(arg) + self.assertEqual(m._astring(once), once) + twice = m._list_mailbox(arg) + self.assertEqual(m._list_mailbox(twice), twice) + + def test_list_mailbox(self): + m = imaplib.IMAP4.__new__(imaplib.IMAP4) + m._encoding = 'ascii' + # Wildcards are not quoted in a list pattern. + self.assertEqual(m._list_mailbox('*'), b'*') + self.assertEqual(m._list_mailbox('%'), b'%') + self.assertEqual(m._list_mailbox('foo/%'), b'foo/%') + # But spaces still require quoting. + self.assertEqual(m._list_mailbox('New folder'), b'"New folder"') + self.assertEqual(m._list_mailbox('"New folder"'), b'"New folder"') + # As do non-ASCII names; wildcards keep their meaning inside a + # quoted string. + m._encoding = 'utf-8' + self.assertEqual(m._list_mailbox('Entwürfe/%'), + '"Entwürfe/%"'.encode()) + if ssl: class SecureTCPServer(socketserver.TCPServer): @@ -279,7 +338,7 @@ def cmd_LOGOUT(self, tag, args): self._send_tagged(tag, 'OK', 'LOGOUT completed') def cmd_LOGIN(self, tag, args): - self.server.logged = args[0] + self.server.logged = args self._send_tagged(tag, 'OK', 'LOGIN completed') def cmd_SELECT(self, tag, args): @@ -510,6 +569,7 @@ def cmd_APPEND(self, tag, args): code, _ = client.enable('UTF8=ACCEPT') self.assertEqual(code, 'OK') self.assertEqual(client._encoding, 'utf-8') + self.assertEqual(server.args, ['UTF8=ACCEPT']) msg_string = 'Subject: üñí©öðé' typ, data = client.append( None, None, None, (msg_string + '\n').encode('utf-8')) @@ -537,6 +597,26 @@ def cmd_AUTHENTICATE(self, tag, args): with self.assertRaisesRegex(imaplib.IMAP4.error, 'charset.*UTF8'): client.search('foo', 'bar') + def test_utf8_mailbox_name(self): + class UTF8Server(SimpleIMAPHandler): + capabilities = 'AUTH ENABLE UTF8=ACCEPT' + def cmd_ENABLE(self, tag, args): + self._send_tagged(tag, 'OK', 'ENABLE successful') + def cmd_AUTHENTICATE(self, tag, args): + self._send_textline('+') + self.server.response = yield + self._send_tagged(tag, 'OK', 'FAKEAUTH successful') + client, server = self._setup(UTF8Server) + typ, _ = client.authenticate('MYAUTH', lambda x: b'fake') + self.assertEqual(typ, 'OK') + typ, _ = client.enable('UTF8=ACCEPT') + self.assertEqual(typ, 'OK') + # A non-ASCII mailbox name is only allowed in a quoted string + # or a literal, never in an atom (RFC 6855). + typ, _ = client.select('Entwürfe') + self.assertEqual(typ, 'OK') + self.assertEqual(server.is_selected, ['"Entwürfe"']) + def test_bad_auth_name(self): class MyServer(SimpleIMAPHandler): def cmd_AUTHENTICATE(self, tag, args): @@ -669,7 +749,7 @@ def test_with_statement(self): _, server = self._setup(SimpleIMAPHandler, connect=False) with self.imap_class(*server.server_address) as imap: imap.login('user', 'pass') - self.assertEqual(server.logged, 'user') + self.assertEqual(server.logged, ['user', '"pass"']) self.assertIsNone(server.logged) def test_with_statement_logout(self): @@ -677,7 +757,7 @@ def test_with_statement_logout(self): _, server = self._setup(SimpleIMAPHandler, connect=False) with self.imap_class(*server.server_address) as imap: imap.login('user', 'pass') - self.assertEqual(server.logged, 'user') + self.assertEqual(server.logged, ['user', '"pass"']) imap.logout() self.assertIsNone(server.logged) self.assertIsNone(server.logged) @@ -752,11 +832,33 @@ def test_idle_delayed_packet(self): self.fail('multi-packet response was corrupted by idle timeout') def test_login(self): - client, _ = self._setup(SimpleIMAPHandler) + client, server = self._setup(SimpleIMAPHandler) typ, data = client.login('user', 'pass') self.assertEqual(typ, 'OK') self.assertEqual(data[0], b'LOGIN completed') self.assertEqual(client.state, 'AUTH') + # The user name is quoted only when necessary, but the password + # is always quoted. + self.assertEqual(server.logged, ['user', '"pass"']) + self.assertRaises(imaplib.IMAP4.error, client.login, 'user', 'pass') + + def test_login_quoted(self): + client, server = self._setup(SimpleIMAPHandler) + typ, data = client.login('us*r', 'p%ss') + self.assertEqual(typ, 'OK') + self.assertEqual(data[0], b'LOGIN completed') + self.assertEqual(client.state, 'AUTH') + self.assertEqual(server.logged, ['"us*r"', '"p%ss"']) + + def test_login_quoted2(self): + # An already quoted user name is passed through unchanged, rather + # than being quoted a second time; the password is always quoted. + client, server = self._setup(SimpleIMAPHandler) + typ, data = client.login('"user"', '"pass"') + self.assertEqual(typ, 'OK') + self.assertEqual(data[0], b'LOGIN completed') + self.assertEqual(client.state, 'AUTH') + self.assertEqual(server.logged, ['"user"', r'"\"pass\""']) def test_append_translate_line_endings(self): # By default line endings are normalized to CRLF; False sends the @@ -785,6 +887,11 @@ def cmd_APPEND(self, tag, args): translate_line_endings=False) self.assertEqual(server.response, message) + # The mailbox is quoted and the flags are wrapped in parentheses + # when necessary. + client.append('New folder', r'\Seen', None, b'data') + self.assertEqual(server.args, ['"New folder"', r'(\Seen)', '{4}']) + def test_login_capabilities(self): # A server may advertise new capabilities after login (as an # untagged CAPABILITY response); imaplib must refresh its cached @@ -873,6 +980,12 @@ def test_lsub(self): self.assertEqual(typ, 'OK') self.assertEqual(server.args, ['~/Mail/', '%']) + # The directory is quoted when necessary; wildcards in the pattern + # are preserved. + typ, data = client.lsub('New folder', '%') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"New folder"', '%']) + def test_extra_blank_line_after_literal(self): # Some buggy servers send an extra blank line after the counted # literal data. imaplib should skip it instead of failing. @@ -940,6 +1053,10 @@ def test_select(self): self.assertEqual(server.is_selected, ['INBOX']) self.assertTrue(client.is_readonly) + typ, data = client.select('New folder') + self.assertEqual(typ, 'OK') + self.assertEqual(server.is_selected, ['"New folder"']) + def test_expunge(self): client, server = self._setup(make_simple_handler('EXPUNGE', ['* 3 EXPUNGE', '* 3 EXPUNGE', '* 5 EXPUNGE', '* 8 EXPUNGE'])) @@ -1022,6 +1139,11 @@ def test_create(self): self.assertEqual(data, [b'CREATE completed']) self.assertEqual(server.args, ['owatagusiam/blurdybloop']) + typ, data = client.create('New folder') + self.assertEqual(typ, 'OK') + self.assertEqual(data, [b'CREATE completed']) + self.assertEqual(server.args, ['"New folder"']) + def test_copy(self): client, server = self._setup(make_simple_handler('COPY')) client.login('user', 'pass') @@ -1031,6 +1153,11 @@ def test_copy(self): self.assertEqual(data, [b'COPY completed']) self.assertEqual(server.args, ['2:4', 'MEETING']) + typ, data = client.copy('2:4', 'New folder') + self.assertEqual(typ, 'OK') + self.assertEqual(data, [b'COPY completed']) + self.assertEqual(server.args, ['2:4', '"New folder"']) + def test_uid_copy(self): client, server = self._setup(make_simple_handler('UID', completed='UID COPY completed')) @@ -1041,6 +1168,11 @@ def test_uid_copy(self): self.assertEqual(data, [None]) self.assertEqual(server.args, ['COPY', '4827313:4828442', 'MEETING']) + typ, data = client.uid('copy', '4827313:4828442', 'New folder') + self.assertEqual(typ, 'OK') + self.assertEqual(data, [None]) + self.assertEqual(server.args, ['COPY', '4827313:4828442', '"New folder"']) + def test_store(self): client, server = self._setup(make_simple_handler('STORE', [ r'* 2 FETCH (FLAGS (\Deleted \Seen))', @@ -1079,6 +1211,10 @@ def test_uid_store(self): ]) self.assertEqual(server.args, ['STORE', '4827313:4828442', '+FLAGS', r'(\Deleted)']) + typ, data = client.uid('store', '4827313:4828442', '+FLAGS', r'\Deleted') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['STORE', '4827313:4828442', '+FLAGS', r'(\Deleted)']) + def test_fetch(self): # The handler expands the requested sequence set and answers for # exactly those messages, so the test exercises the round trip of @@ -1123,6 +1259,19 @@ def cmd_FETCH(self, tag, args): self.assertEqual(data, [br'1 (FLAGS (\Seen))']) self.assertEqual(server.args, ['1', '(BODY[HEADER.FIELDS (DATE FROM)])']) + # message_parts is wrapped in parentheses if it is not already. + typ, data = client.fetch('2:4', 'FLAGS') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['2:4', '(FLAGS)']) + + # But the macros are not, as they are not data item names. + typ, data = client.fetch('2:4', 'ALL') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['2:4', 'ALL']) + typ, data = client.fetch('2:4', 'fast') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['2:4', 'fast']) + def test_uid_fetch(self): client, server = self._setup(make_simple_handler('UID', [ r'* 23 FETCH (FLAGS (\Seen) UID 4827313)', @@ -1140,6 +1289,14 @@ def test_uid_fetch(self): ]) self.assertEqual(server.args, ['FETCH', '4827313:4828442', '(FLAGS)']) + typ, data = client.uid('fetch', '4827313:4828442', 'FLAGS') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['FETCH', '4827313:4828442', '(FLAGS)']) + + typ, data = client.uid('fetch', '4827313:4828442', 'ALL') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['FETCH', '4827313:4828442', 'ALL']) + def test_partial(self): client, server = self._setup(make_simple_handler('PARTIAL', ['* 1 FETCH (RFC822.TEXT<0.10> "0123456789")'])) @@ -1173,6 +1330,10 @@ def test_search(self): self.assertEqual(data, [b'43']) self.assertEqual(server.args, ['CHARSET', 'UTF-8', 'TEXT', 'XXXXXX']) + typ, data = client.search('NF_Z_62-010_(1973)', 'TEXT', 'XXXXXX') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['CHARSET', '"NF_Z_62-010_(1973)"', 'TEXT', 'XXXXXX']) + def test_uid_search(self): response = [] client, server = self._setup(make_simple_handler('UID', response, @@ -1224,6 +1385,10 @@ def test_sort(self): self.assertEqual(data, [br'']) self.assertEqual(server.args, ['(SUBJECT)', 'US-ASCII', 'TEXT', '"not in mailbox"']) + typ, data = client.sort('SUBJECT', 'NF_Z_62-010_(1973)', 'TEXT', '"not in mailbox"') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['(SUBJECT)', '"NF_Z_62-010_(1973)"', 'TEXT', '"not in mailbox"']) + def test_uid_sort(self): response = [] client, server = self._setup(make_simple_handler('UID', response, @@ -1248,6 +1413,10 @@ def test_uid_sort(self): self.assertEqual(data, [br'']) self.assertEqual(server.args, ['SORT', '(SUBJECT)', 'US-ASCII', 'TEXT', '"not in mailbox"']) + typ, data = client.uid('sort', 'SUBJECT', 'NF_Z_62-010_(1973)', 'TEXT', '"not in mailbox"') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['SORT', '(SUBJECT)', '"NF_Z_62-010_(1973)"', 'TEXT', '"not in mailbox"']) + def test_thread(self): response = [] client, server = self._setup(make_simple_handler('THREAD', response)) @@ -1285,6 +1454,10 @@ def test_thread(self): b'(199)(200 202)(201)(203)(204)(205 206 207)(208)']) self.assertEqual(server.args, ['ORDEREDSUBJECT', 'US-ASCII', 'TEXT', '"gewp"']) + typ, data = client.thread('ORDEREDSUBJECT', 'NF_Z_62-010_(1973)', 'TEXT', '"gewp"') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['ORDEREDSUBJECT', '"NF_Z_62-010_(1973)"', 'TEXT', '"gewp"']) + def test_uid_thread(self): response = [] client, server = self._setup(make_simple_handler('UID', response, @@ -1323,6 +1496,10 @@ def test_uid_thread(self): b'(199)(200 202)(201)(203)(204)(205 206 207)(208)']) self.assertEqual(server.args, ['THREAD', 'ORDEREDSUBJECT', 'US-ASCII', 'TEXT', '"gewp"']) + typ, data = client.uid('THREAD', 'ORDEREDSUBJECT', 'NF_Z_62-010_(1973)', 'TEXT', '"gewp"') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['THREAD', 'ORDEREDSUBJECT', '"NF_Z_62-010_(1973)"', 'TEXT', '"gewp"']) + def test_delete(self): client, server = self._setup(make_simple_handler('DELETE')) client.login('user', 'pass') @@ -1335,6 +1512,10 @@ def test_delete(self): self.assertEqual(typ, 'OK') self.assertEqual(server.args, ['foo/bar']) + typ, data = client.delete('New folder') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"New folder"']) + def test_rename(self): client, server = self._setup(make_simple_handler('RENAME')) client.login('user', 'pass') @@ -1343,6 +1524,10 @@ def test_rename(self): self.assertEqual(data, [b'RENAME completed']) self.assertEqual(server.args, ['blurdybloop', 'sarasoop']) + typ, data = client.rename('Old folder', 'New folder') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"Old folder"', '"New folder"']) + def test_subscribe(self): client, server = self._setup(make_simple_handler('SUBSCRIBE')) client.login('user', 'pass') @@ -1351,6 +1536,10 @@ def test_subscribe(self): self.assertEqual(data, [b'SUBSCRIBE completed']) self.assertEqual(server.args, ['#news.comp.mail.mime']) + typ, data = client.subscribe('New folder') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"New folder"']) + def test_unsubscribe(self): client, server = self._setup(make_simple_handler('UNSUBSCRIBE')) client.login('user', 'pass') @@ -1359,6 +1548,10 @@ def test_unsubscribe(self): self.assertEqual(data, [b'UNSUBSCRIBE completed']) self.assertEqual(server.args, ['#news.comp.mail.mime']) + typ, data = client.unsubscribe('New folder') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"New folder"']) + def test_list(self): client, server = self._setup(make_simple_handler('LIST', [r'* LIST (\Noselect) "/" ""', @@ -1374,6 +1567,15 @@ def test_list(self): self.assertEqual(typ, 'OK') self.assertEqual(server.args, ['~/Mail/', '%']) + typ, data = client.list('New folder', '*') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"New folder"', '*']) + + # A pattern without wildcards is quoted when necessary. + typ, data = client.list('~/Mail/', 'My Folder') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['~/Mail/', '"My Folder"']) + def test_status(self): client, server = self._setup(make_simple_handler('STATUS', ['* STATUS blurdybloop (MESSAGES 231 UIDNEXT 44292)'])) @@ -1383,6 +1585,11 @@ def test_status(self): self.assertEqual(data, [b'blurdybloop (MESSAGES 231 UIDNEXT 44292)']) self.assertEqual(server.args, ['blurdybloop', '(UIDNEXT MESSAGES)']) + # The names argument is wrapped in parentheses if it is not already. + typ, data = client.status('New folder', 'UIDNEXT MESSAGES') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"New folder"', '(UIDNEXT MESSAGES)']) + def test_getacl(self): client, server = self._setup(make_simple_handler('GETACL', ['* ACL INBOX Fred rwipslxetad'])) @@ -1392,6 +1599,10 @@ def test_getacl(self): self.assertEqual(data, [b'INBOX Fred rwipslxetad']) self.assertEqual(server.args, ['INBOX']) + typ, data = client.getacl('New folder') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"New folder"']) + def test_setacl(self): client, server = self._setup(make_simple_handler('SETACL')) client.login('user', 'pass') @@ -1400,6 +1611,15 @@ def test_setacl(self): self.assertEqual(data, [b'SETACL completed']) self.assertEqual(server.args, ['INBOX', 'Fred', 'rwipslxetad']) + typ, data = client.setacl('New folder', 'Fred', '+lr') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"New folder"', 'Fred', '+lr']) + + # The identifier and the rights are quoted when necessary too. + typ, data = client.setacl('INBOX', 'John Doe', 'a b') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['INBOX', '"John Doe"', '"a b"']) + def test_deleteacl(self): client, server = self._setup(make_simple_handler('DELETEACL')) client.login('user', 'pass') @@ -1408,6 +1628,11 @@ def test_deleteacl(self): self.assertEqual(data, [b'DELETEACL completed']) self.assertEqual(server.args, ['INBOX', 'Fred']) + # The identifier is quoted when necessary too. + typ, data = client.deleteacl('New folder', 'John Doe') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"New folder"', '"John Doe"']) + def test_myrights(self): client, server = self._setup(make_simple_handler('MYRIGHTS', ['* MYRIGHTS INBOX rwiptsldaex'])) @@ -1417,6 +1642,10 @@ def test_myrights(self): self.assertEqual(data, [b'INBOX rwiptsldaex']) self.assertEqual(server.args, ['INBOX']) + typ, data = client.myrights('New folder') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"New folder"']) + def test_getquota(self): client, server = self._setup(make_simple_handler('GETQUOTA', ['* QUOTA "" (STORAGE 10 512)'])) @@ -1426,6 +1655,10 @@ def test_getquota(self): self.assertEqual(data, [b'"" (STORAGE 10 512)']) self.assertEqual(server.args, ['#news']) + typ, data = client.getquota('') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['""']) + def test_getquotaroot(self): client, server = self._setup(make_simple_handler('GETQUOTAROOT', ['* QUOTAROOT INBOX ""', '* QUOTA "" (STORAGE 10 512)'])) @@ -1435,6 +1668,10 @@ def test_getquotaroot(self): self.assertEqual(data, [[b'INBOX ""'], [b'"" (STORAGE 10 512)']]) self.assertEqual(server.args, ['INBOX']) + typ, data = client.getquotaroot('New folder') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"New folder"']) + def test_setquota(self): client, server = self._setup(make_simple_handler('SETQUOTA', ['* QUOTA "" (STORAGE 512)'])) @@ -1444,6 +1681,15 @@ def test_setquota(self): self.assertEqual(data, [b'"" (STORAGE 512)']) self.assertEqual(server.args, ['#news', '(STORAGE 512)']) + typ, data = client.setquota('', '(STORAGE 512)') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['""', '(STORAGE 512)']) + + # The limits argument is wrapped in parentheses if it is not already. + typ, data = client.setquota('', 'STORAGE 512') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['""', '(STORAGE 512)']) + def test_getannotation(self): client, server = self._setup(make_simple_handler('GETANNOTATION', ['* ANNOTATION INBOX "/comment" ("value.shared" "Hello")'])) @@ -1453,6 +1699,10 @@ def test_getannotation(self): self.assertEqual(data, [b'INBOX "/comment" ("value.shared" "Hello")']) self.assertEqual(server.args, ['INBOX', '/comment', 'value.shared']) + typ, data = client.getannotation('New folder', '/comment', 'value.shared') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"New folder"', '/comment', 'value.shared']) + def test_setannotation(self): client, server = self._setup(make_simple_handler('SETANNOTATION')) client.login('user', 'pass') @@ -1462,6 +1712,12 @@ def test_setannotation(self): self.assertEqual(server.args, ['INBOX', '/comment', '("value.shared" "My comment")']) + typ, data = client.setannotation('New folder', '/comment', + '("value.shared" "My comment")') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, + ['"New folder"', '/comment', '("value.shared" "My comment")']) + def test_proxyauth(self): client, server = self._setup(make_simple_handler('PROXYAUTH')) client.login('user', 'pass') @@ -1470,6 +1726,10 @@ def test_proxyauth(self): self.assertEqual(data, [b'PROXYAUTH completed']) self.assertEqual(server.args, ['user']) + typ, data = client.proxyauth('us er') + self.assertEqual(typ, 'OK') + self.assertEqual(server.args, ['"us er"']) + def test_xatom(self): client, server = self._setup(make_simple_handler('MYCOMMAND', completed='MYCOMMAND completed')) @@ -1480,6 +1740,16 @@ def test_xatom(self): self.assertEqual(data, [b'MYCOMMAND completed']) self.assertEqual(server.args, ['arg1', 'arg2']) + def test_uppercase_command_names(self): + client, server = self._setup(SimpleIMAPHandler) + client.login('user', 'pass') + self.assertEqual(client.CAPABILITY, client.capability) + self.assertEqual(client.SELECT, client.select) + typ, data = client.CAPABILITY() + self.assertEqual(typ, 'OK') + with self.assertRaises(AttributeError): + client.NONEXISTENT + def test_control_characters(self): client, _ = self._setup(SimpleIMAPHandler) for c0 in support.control_characters_c0(): @@ -1753,6 +2023,7 @@ def cmd_APPEND(self, tag, args): code, _ = client.enable('UTF8=ACCEPT') self.assertEqual(code, 'OK') self.assertEqual(client._encoding, 'utf-8') + self.assertEqual(server.args, ['UTF8=ACCEPT']) msg_string = 'Subject: üñí©öðé' typ, data = client.append( None, None, None, (msg_string + '\n').encode('utf-8')) @@ -1908,7 +2179,7 @@ def test_with_statement(self): with self.reaped_server(SimpleIMAPHandler) as server: with self.imap_class(*server.server_address) as imap: imap.login('user', 'pass') - self.assertEqual(server.logged, 'user') + self.assertEqual(server.logged, ['user', '"pass"']) self.assertIsNone(server.logged) @threading_helper.reap_threads @@ -1917,7 +2188,7 @@ def test_with_statement_logout(self): with self.reaped_server(SimpleIMAPHandler) as server: with self.imap_class(*server.server_address) as imap: imap.login('user', 'pass') - self.assertEqual(server.logged, 'user') + self.assertEqual(server.logged, ['user', '"pass"']) imap.logout() self.assertIsNone(server.logged) self.assertIsNone(server.logged) diff --git a/Lib/test/test_tkinter/test_simpledialog.py b/Lib/test/test_tkinter/test_simpledialog.py index 5c739d9ad6e6422..33a0173ba67adb9 100644 --- a/Lib/test/test_tkinter/test_simpledialog.py +++ b/Lib/test/test_tkinter/test_simpledialog.py @@ -44,6 +44,7 @@ def test_use_ttk(self): self.assertEqual(str(d.root.cget('background')), ttk.Style(d.root).lookup('.', 'background')) # The bindings work with the themed buttons too. + self.require_mapped(d.root) d._buttons[0].focus_force() d.root.update() d.root.event_generate('') @@ -139,6 +140,7 @@ def test_alt_key(self): # Alt + an underlined character (the "underline" button option) invokes # the matching button (cf. tk::AmpWidget in tk::MessageBox). d = self.create(buttons=['Yes', {'text': 'No', 'underline': 0}]) + self.require_mapped(d.root) d._buttons[0].focus_force() d.root.update() d.root.event_generate('') # "No" -> underline 0 -> "N" @@ -149,6 +151,7 @@ def test_return_invokes_focused_button(self): # invokes the button with the focus, even if it is not the # default and the focus was not moved by keyboard traversal. d = self.create(buttons=['Yes', 'No']) # default 0 + self.require_mapped(d.root) d._buttons[1].focus_force() d.root.update() d.root.event_generate('') @@ -158,6 +161,7 @@ def test_return_invokes_focused_button(self): def test_focus_next_then_return(self): # moves the focus to the next button; invokes it. d = self.create(buttons=['Yes', 'No']) + self.require_mapped(d.root) d._buttons[0].focus_force() d.root.update() d._buttons[0].event_generate('') @@ -169,6 +173,7 @@ def test_focus_next_then_return(self): def test_focus_prev_then_return(self): # moves the focus to the previous button. d = self.create(buttons=['Yes', 'No']) + self.require_mapped(d.root) d._buttons[1].focus_force() d.root.update() d._buttons[1].event_generate('') @@ -180,6 +185,7 @@ def test_focus_prev_then_return(self): def test_return_activates_default(self): # with the focus off the buttons invokes the default button. d = self.create() # default 0 + self.require_mapped(d.root) d.root.focus_force() # the dialog, not a button, has the focus d.root.update() d.root.event_generate('') @@ -190,6 +196,7 @@ def test_return_no_default(self): # With no default button, off the buttons rings the bell and # leaves the dialog open instead of activating a button. d = self.create(default=None) + self.require_mapped(d.root) d.root.focus_force() # the dialog, not a button, has the focus d.root.update() bells = [] diff --git a/Lib/urllib/robotparser.py b/Lib/urllib/robotparser.py index e70eae800367840..0c3e5d928909358 100644 --- a/Lib/urllib/robotparser.py +++ b/Lib/urllib/robotparser.py @@ -65,9 +65,17 @@ def read(self): f = urllib.request.urlopen(self.url) except urllib.error.HTTPError as err: if err.code in (401, 403): + # If access to robot.txt has the status Unauthorized/Forbidden, + # then most likely this applies to the entire site. self.disallow_all = True - elif err.code >= 400 and err.code < 500: + elif 400 <= err.code < 500: + # RFC 9309, Section 2.3.1.3: the crawler MAY access any + # resources on the server. self.allow_all = True + elif 500 <= err.code < 600: + # RFC 9309, Section 2.3.1.4: the crawler MUST assume + # complete disallow. + self.disallow_all = True err.close() else: raw = f.read() diff --git a/Misc/NEWS.d/next/Library/2025-09-05-20-50-35.gh-issue-79638.Y-JfaH.rst b/Misc/NEWS.d/next/Library/2025-09-05-20-50-35.gh-issue-79638.Y-JfaH.rst new file mode 100644 index 000000000000000..bd9fff0bc2e31b6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2025-09-05-20-50-35.gh-issue-79638.Y-JfaH.rst @@ -0,0 +1,2 @@ +Disallow all access in :mod:`urllib.robotparser` if the ``robots.txt`` file +is unreachable due to server or network errors. diff --git a/Misc/NEWS.d/next/Library/2026-06-30-12-00-00.gh-issue-40038.qK7mGv.rst b/Misc/NEWS.d/next/Library/2026-06-30-12-00-00.gh-issue-40038.qK7mGv.rst new file mode 100644 index 000000000000000..1f393d23266bcef --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-06-30-12-00-00.gh-issue-40038.qK7mGv.rst @@ -0,0 +1,6 @@ +:mod:`imaplib` now again quotes command arguments when necessary, for +example mailbox names containing a space. Such quoting was inadvertently +disabled when the module was ported to Python 3, and the arguments are now +quoted according to the :rfc:`3501` grammar. For backward compatibility, +an argument already enclosed in double quotes is left unchanged, so code +that quotes arguments itself keeps working. diff --git a/Misc/NEWS.d/next/Library/2026-07-03-21-10-00.gh-issue-54930.hq09Er.rst b/Misc/NEWS.d/next/Library/2026-07-03-21-10-00.gh-issue-54930.hq09Er.rst new file mode 100644 index 000000000000000..1cfc0212c609a69 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-07-03-21-10-00.gh-issue-54930.hq09Er.rst @@ -0,0 +1,5 @@ +Error responses of :class:`http.server.BaseHTTPRequestHandler` to malformed +request lines now include a status line and headers instead of being sent in +the bare HTTP/0.9 style. +Only a valid HTTP/0.9 request (a two-word ``GET`` request line) now receives +an HTTP/0.9 style response.