[red-knot] Understand `typing.Callable` #16493

dhruvmanila · 2025-03-04T09:16:33Z

Summary

This PR implements a general callable type that wraps around a Signature and it uses that new type to represent typing.Callable.

It also implements Display support for Callable. The format is as:

([<arg name>][: <arg type>][ = <default type>], ...) -> <return type>

The / and * separators are added at the correct boundary for positional-only and keyword-only parameters. Now, as typing.Callable only has positional-only parameters, the rendered signature would be:

Callable[[int, str], None]
# (int, str, /) -> None

The / separator represents that all the arguments are positional-only.

The relationship methods that check assignability, subtype relationship, etc. are not yet implemented and will be done so as a follow-up.

Test Plan

Add test cases for display support for Signature and various mdtest for typing.Callable.

github-actions · 2025-03-05T10:44:19Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

crates/red_knot_python_semantic/src/types/infer.rs

carljm

Very cool! Great work.

crates/red_knot_python_semantic/resources/mdtest/annotations/callable.md

carljm · 2025-03-05T18:30:32Z

crates/red_knot_python_semantic/resources/mdtest/annotations/callable.md

+    # TODO: This should be `() -> Unknown` which requires understanding that `int` is not a param spec variable
+    reveal_type(c)  # revealed: (*args: @Todo(todo signature *args), **kwargs: @Todo(todo signature **kwargs)) -> str


I actually think inferring Callable[int, str] (or Callable[42, str] below, or any similar malformation) as equivalent to Callable[..., str] (and of course also emitting an error) is probably better, since it will minimize cascading false positives. If you meant to say Callable[[int], str] but you accidentally said Callable[int, str], ideally we would not emit a false positive on every subsequent attempt to call that callable with one argument.

Yeah, makes sense. Just want to confirm that we still do want to keep the return type intact i.e., use Callable[..., str] and not Callable[..., Unknown] which is what I've pushed but happy to change it back to using Unknown.

I would prefer Callable[..., Unknown] here for the same reasons I stated in #16493 (comment) — it just seems like a simpler policy that involves less guesswork and that will be much easier to explain to users

Ok, I think I'll also use Callable[..., Unknown] for type forms with more than 2 arguments for similar reasons and the fact that we're at least sure that it's a callable object.

crates/red_knot_python_semantic/resources/mdtest/annotations/callable.md

carljm · 2025-03-05T18:35:57Z

crates/red_knot_python_semantic/resources/mdtest/annotations/callable.md

+Or, when it's a literal type:
+
+```py
+# error: [invalid-type-form] "The first argument to `typing.Callable` must be either a list of types, parameter specification, `typing.Concatenate`, or `...`"


nit: "ParamSpec" is probably clearer than "parameter specification", since it's a term that can be easily searched in the typing spec / docs -- the latter could be understand to have a broader meaning

Yeah, I did a quick search on typing and Python docs and it seems that "parameter specification" would not give users much useful information. I'll change it to use typing.ParamSpec

For context, I mainly used this because ParamSpec itself cannot be used directly as type expression similar to TypeVar so suggesting ParamSpec could be slightly confused by whether to directly use ParamSpec v/s a variable that represents the parameter specification.

crates/red_knot_python_semantic/src/types/infer.rs

carljm · 2025-03-05T20:14:16Z

crates/red_knot_python_semantic/src/types/signatures.rs

@@ -92,6 +101,23 @@ impl<'db> Parameters<'db> {
        ])
    }

+    /// Return parameters that accepts any arguments i.e., `(*args: Any, **kwargs: Any)`
+    pub(crate) fn any() -> Self {
+        // TODO: Should this be represented using `(*Any, **Any)` instead?


Oh, yeah, maybe we should ideally omit the names here? Not sure how much fallout that change would have

Now that we've decided to use (...) -> Unknown I don't think this is relevant so I've avoided this.

crates/red_knot_python_semantic/src/types/type_ordering.rs

MichaReiser · 2025-03-06T07:42:53Z

crates/red_knot_python_semantic/src/types.rs

@@ -2664,6 +2699,10 @@ impl<'db> Type<'db> {
            Type::Callable(CallableType::WrapperDescriptorDunderGet) => {
                KnownClass::WrapperDescriptorType.to_class_literal(db)
            }
+            Type::Callable(CallableType::General(_)) => {
+                // TODO


Can we add a small explanation of what needs to be done here or remove the comment if it's redundant to the todo_type description

I'll expand the comment. I avoided here as it would be similar to todo_type but I still grep for TODO comments 😅

crates/red_knot_python_semantic/src/types.rs

crates/red_knot_python_semantic/src/types/display.rs

dhruvmanila · 2025-03-06T12:04:14Z

Post review changes

Use (...) -> Unknown consistently
Display ... for gradual form. This is done using an additional flag on Parameters to indicate the gradual form.
Use Any for gradual form while Unknown for invalid forms
Avoid special casing for param spec variable
Don't error on bare Callable, infer it as (...) -> Unknown
Remove various todos, add a couple more
Improve display handling of Signature

Regarding the relation implementation, I think I'd prefer to add it in a follow-up PR to keep the changes separate and easier to review.

dhruvmanila added the red-knot Multi-file analysis & type inference label Mar 4, 2025

dhruvmanila force-pushed the dhruv/callable-type branch from e71d87b to dbae500 Compare March 5, 2025 08:44

[red-knot] Understand typing.Callable

05a7a5b

dhruvmanila force-pushed the dhruv/callable-type branch from dbae500 to 05a7a5b Compare March 5, 2025 10:35

dhruvmanila added 4 commits March 5, 2025 16:17

fix pre-commit

12c8d2e

Move invalid syntax test case

dec9d49

Avoid todo!() macro which panics

ff7d3ce

remove unused method

0ae7e87

dhruvmanila commented Mar 5, 2025

View reviewed changes

crates/red_knot_python_semantic/src/types/infer.rs Outdated Show resolved Hide resolved

crates/red_knot_python_semantic/src/types/infer.rs Outdated Show resolved Hide resolved

crates/red_knot_python_semantic/src/types/infer.rs Outdated Show resolved Hide resolved

dhruvmanila marked this pull request as ready for review March 5, 2025 17:37

dhruvmanila requested review from carljm, MichaReiser, AlexWaygood and sharkdp as code owners March 5, 2025 17:38

MichaReiser mentioned this pull request Mar 5, 2025

[red-knot] Avoid running ecosystem checks when not necessary #16522

Open

carljm reviewed Mar 5, 2025

View reviewed changes

MichaReiser approved these changes Mar 6, 2025

View reviewed changes

dhruvmanila added 6 commits March 6, 2025 14:26

Address review feedback

89a33fb

Impl union element ordering

2b9ee87

continue addressing feedback

1d9a6b1

Avoid invalid type form error for literals

44aec36

fix pre-commit

a23220f

update docs

926fa8e

Use (...) -> Unknown consistently

e8c9327

dhruvmanila requested a review from carljm March 6, 2025 15:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[red-knot] Understand `typing.Callable` #16493

[red-knot] Understand `typing.Callable` #16493

dhruvmanila commented Mar 4, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

carljm left a comment

carljm Mar 5, 2025

dhruvmanila Mar 6, 2025

AlexWaygood Mar 6, 2025

dhruvmanila Mar 6, 2025

carljm Mar 5, 2025

dhruvmanila Mar 6, 2025

dhruvmanila Mar 6, 2025

carljm Mar 5, 2025

dhruvmanila Mar 6, 2025

MichaReiser Mar 6, 2025

dhruvmanila Mar 6, 2025

dhruvmanila commented Mar 6, 2025 •

edited

Loading

		# TODO: This should be `() -> Unknown` which requires understanding that `int` is not a param spec variable
		reveal_type(c) # revealed: (args: @Todo(todo signature args), kwargs: @Todo(todo signature kwargs)) -> str

[red-knot] Understand typing.Callable #16493

Are you sure you want to change the base?

[red-knot] Understand typing.Callable #16493

Conversation

dhruvmanila commented Mar 4, 2025 • edited Loading

Summary

Test Plan

github-actions bot commented Mar 5, 2025 • edited Loading

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

carljm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhruvmanila commented Mar 6, 2025 • edited Loading

Post review changes

[red-knot] Understand `typing.Callable` #16493

[red-knot] Understand `typing.Callable` #16493

dhruvmanila commented Mar 4, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

`ruff-ecosystem` results

dhruvmanila commented Mar 6, 2025 •

edited

Loading