Optional class and protocol fields and methods

[srittau]

Sometimes, implementations check for the existence of a method on an object before using it. Take this example from Python 2.7's urllib:

class addbase:
    def __init__(self, fp):
        self.fp = fp
        self.read = self.fp.read
        self.readline = self.fp.readline
        if hasattr(self.fp, "readlines"): self.readlines = self.fp.readlines
        if hasattr(self.fp, "fileno"):
            self.fileno = self.fp.fileno
        else:
            self.fileno = lambda: None
        if hasattr(self.fp, "__iter__"):
            self.__iter__ = self.fp.__iter__
            if hasattr(self.fp, "next"):
                self.next = self.fp.next

Currently this is best modeled by leaving these methods out from a protocol. But this means that their signature is unknown, and mypy will complain about the non-existing attribute. It would be useful to be able to model this somehow, for example by using a decorator for such methods.

[ilevkivskyi]

This was explicitly deferred in PEP 544. On the other hand this would be not hard to implement now. Also another structural type in mypy -- TypedDict -- supports this. So maybe we should also support total=False for protocols?

@JukkaL what do you think?

[srittau]

I don't think a total=False attribute would be sufficient. In the example above - in fact in all cases where I have encountered this so far - there are required members as well as optional ones. A per-attribute flag (a decorator?) would serve these cases much better.

[ilevkivskyi]

I don't think a total=False attribute would be sufficient

It is always sufficient. Whether it is convenient is a different question.

[srittau]

I mentioned this again on typing-sig. It could reuse PEP 655's NotRequired class:

class MyProto:
    x: NotRequired[int]
    @not_required
    def foo(self) -> None: ...

It would also make sense for non-protocols:

class Foo:
    x: NotRequired[int]
    def setup(self) -> None:
        self.x = some_calc()

(Not a design I would recommend, but I have seen this from time to time in the wild.)

[vnmabus]

I think this could be helpful for solving #1498 by marking _my_property as a non-required attribute. What do you think?

[davetapley]

See also:

• Type Hints: Add type hints to tutorials, snippets and examples falconry/falcon#1820 (comment)

[ottokruse]

It's 2024 and I need this for a scenario where I want to type classes that may optionally implement a method.

My function takes in instances that should match a protocol, where one method (reset()) is optional. If they have defined it, my function will call it, if not proceed straight with other logic.

class MyClassProtocol(Protocol):
  pass
  # Can't put this in or that'd make it mandatory:
  # def reset() -> void: ...
def my_func(i: MyClassProtocol):
  if hasattr(i, "reset"):
    i.reset() # fails typecheck now
  # other logic

[erictraut]

@ottokruse, here's a solution that might work for you:

Code sample in pyright playground

from typing import Protocol, runtime_checkable

@runtime_checkable
class MyClassProtocol1(Protocol):
    def required_method(self) -> None:
        ...

@runtime_checkable
class MyClassProtocol2(MyClassProtocol1, Protocol):
    def reset(self) -> None:
        ...

type MyClassProtocol = MyClassProtocol1 | MyClassProtocol2

def my_func(i: MyClassProtocol):
    if isinstance(i, MyClassProtocol2):
        i.reset()

# Test with some concrete classes
class Foo:
    def required_method(self) -> None:
        ...

class Bar(Foo):
    def reset(self) -> None:
        ...

my_func(Foo())
my_func(Bar())

[ottokruse]

Thank you Eric!

Also found your typeguard PEP now. Nice work

[yangdanny97]

I did some number crunching here: https://github.com/yangdanny97/type_coverage_py/blob/hasattr_getattr/package_report.json

and it looks like 65% of the top 2000 packages on pypi have calls to hasattr or three-argument calls to getattr in their source code, which suggests that this is a common pattern (unsurprising since these are builtins).

IMO it's common enough in existing code to warrant a PEP for this.

It would also make sense for non-protocols:

class Foo:
    x: NotRequired[int]
    def setup(self) -> None:
        self.x = some_calc()

(Not a design I would recommend, but I have seen this from time to time in the wild.)

I'd advocate for NotRequired to be allowed in classes directly, so that it can be adopted with minimal effort by more people.

If this was protocol-only, I get the sense that maintainers will grumble about writing extra classes just to satisfy the typechecker, and I worry it would be like Callable protocols where it's kind of verbose and relatively harder to adopt.

[yangdanny97]

I came up with some proposed semantics for this - feedback would be greatly appreciated

Semantics

The NotRequired qualifier will be allowed for attribute annotations in classes and protocols. It won't be possible to annotate methods with a decorator equivalent, since there is no way to declare a method without it being bound in the class.

Structural Typechecking

If a protocol has a NotRequired attribute, structural typechecking will depend on finality:

• If the class is final, it can omit the attribute, declare it normally, or declare it with NotRequired
• If the class is not final, it must declare the attribute normally or with NotRequired; the attribute may NOT be omitted

class Proto(Protocol):
    x: NotRequired[int]
class Class1:
    pass
@final
class Class2:
    pass
class Class3:
    x: NotRequired[int]
class Class4:
    x: int
a: Proto = Class1()  # not OK
b: Proto = Class2()  # OK
c: Proto = Class3()  # OK
d: Proto = Class4()  # OK

The requirement that implementing classes must declare the attribute as NotRequired (instead of allowing it to be omitted) is motivated by this example from @JelleZijlstra

class P(Protocol):
    a: int
    b: NotRequired[str]

def user(p: P) -> None:
    if hasattr(p, 'b'):
        print(len(p.b))

class A:
    a: int
    def __init__(self) -> None:
        self.a = 3

class B(A):
    b: int
    def __init__(self) -> None:
        super().__init__()
        self.b = 1

def accept_a(a: A) -> None:
    user(a) # OK, A has an attribute a of the right type and doesn't have an attribute b

accept_a(B()) # boom

@runtime_checkable

Attributes annotated with NotRequired in a runtime_checkable protocol will be skipped when checking an object at runtime. The current runtime checkable behavior only checks for the presence of an attribute with the right name without checking the type.

Overrides

Subclasses may only remove the NotRequired qualifier from an overridden attribute.

Final

NotRequired will not be compatible with the Final qualifier in the same type annotation, since attributes with the latter are required to be initialized so NotRequired wouldn’t do anything.

ReadOnly

PEP767 may introduce read-only attributes. Subclasses will be allowed to override ReadOnly attributes to remove NotRequired (making a non-required attribute required).

class Class1:
    x: ReadOnly[NotRequired[int]]
class Class2(Class1):
    x: ReadOnly[int]  # OK
class Class3(Class1):
    x: ReadOnly[NotRequired[int]]  # OK

Pattern Matching

Not-required attributes are not allowed to be used with __match_args__, and may not be matched in general.

Assignment/Deletion

There will be no changes to assignment or deletion behavior at runtime. For the purposes of typechecking, assignment to NotRequired attributes will work the same as attributes annotated without the qualifier.

Currently, despite all attributes being “required”, none of the major typecheckers prevent attributes from being deleted. This behavior will stay the same, and both regular and NotRequired attributes will be deletable.

Access

There will be no changes to access behavior at runtime. Typecheckers may error if the attribute is accessed without being narrowed (using a hasattrcall or assigning to it) or if getattris used without a default.

This would be similar to emitting errors for accessing non-required TypedDict keys, and narrowing would work the same way (Mypy and Pyre don’t support this kind of error/narrowing, but Pyright does).

Given the lack of standardization of the equivalent typechecking behavior for TypedDicts, I think we probably want to make this behavior optional for now.

class Class1:
    x: NotRequired[int]
c = Class1()
if hasattr(c, "x"):
    c.x  # OK
c.x  # not OK (optional)

Uninitialized Attribute Checks

Typecheckers including Mypy, Pyright, and Pyre, can check for uninitialized attributes, though this is generally a best-effort check and opt-in/experimental in some cases. Typecheckers should not raise an error if the uninitialized attribute is annotated with NotRequired.

Effect for Users

When implementing all the required proposed features as described above (but none of the optional ones), this is what changes.

Compared to leaving the attribute unannotated:

• We know what type the attribute is when accessing/assigning (instead of giving an unknown attribute error/defaulting to Any)
• We can restrict the type of the attribute in a subclass

Compared to annotating the attribute with an unqualified type:

• No typechecker complaints on uninitialized attributes
• The presence of the qualifier documents that the attribute may be absent

Things the proposed semantics do NOT guarantee:

• Absence of NotRequired means that the attribute is present
• Attribute accesses to NotRequired attributes must be gated
• Banning deletion of required attributes

cc @migeed-z @stroxler @samwgoldman @rchen152 @grievejia