Avoid macros and static inline functions

[encukou]

All functions should be exported as proper symbols, so they are usable without a C preprocessor/parser.

We expect a lot of exceptions, which should be added as alternatives to proper functions (as per #4).

Macros or static inline functions can override/shadow a function with the same name and behavior. The dance for that is:

Header file:

static inline returntype
_Py_Foo_impl(args)
{
    ...
}

PyAPI_FUNC(returntype) Py_Foo (args);

// speedup for C/C++
#define Py_Foo _Py_Foo_impl

.c file:

// at the end (after all calls to Py_Foo):
#undef Py_Foo

returntype
Py_Foo (args)
{
    return _Py_Foo_impl(args);
}

Let's have Argument Clinic generate the boilerplate?

---

Another exception are C-specific helper macros, like Py_VISIT or Py_MIN. IMO, the docs for these should document the internals clearly enough that people can rewrite them in other languages.
Unless they're not useful in other languages (like Py_UNREACHABLE or Py_UNUSED).

[vstinner]

See also issue #37: Declare constants as variables, instead of macros.

[pitrou]

I'm not sure why inline functions should be avoided. As long as you make sure there is a corresponding extern declaration somewhere in the compilation units, the symbol should be exposed.

According to https://en.cppreference.com/w/c/language/inline (emphasis mine):

If a function is declared inline in some translation units, it does not need to be declared inline everywhere: at most one translation unit may also provide a regular, non-inline non-static function, or a function declared extern inline. This one translation unit is said to provide the external definition.

There is a helpful StackOverflow answer about this:
https://stackoverflow.com/a/17505846

So you can have this in a .h:

inline returntype
Py_Foo(args)
{
    ...
}

and then in a single .c file (perhaps one dedicated to the extern definitions of public inline functions):

extern PyAPI_FUNC(returntype)
Py_Foo(args);

[gvanrossum]

I'm guessing the (only? by far most common?) use of inline functions in CPython is static inline, and I don't think you can add an extern declaration for those. Possibly the issue title should just be updated to say "static inline" instead of "inline"?

[pitrou]

@gvanrossum I think you're right!

[vstinner]

FYI the rationale for using static inline vs inline was decided when the Py_INCREF() macro was converted to a static inline function: python/cpython#79240 especially this comment: python/cpython#79240 (comment) It was also discussed there to use macros to "always inline" static inline functions (at the end, it was decided to not use that).

The impact on performance of static inline functions (versus macros) was measured in PEP 670 – Convert macros to functions in the Python C API.

[vstinner]

Macros or static inline functions can override/shadow a function with the same name and behavior.

An alternative is to convert static inline functions to regular functions.

So far, I didn't see any conclusive analysis that static inline functions are really faster than regular function calls, especially in third party C extensions.

[pitrou]

FYI the rationale for using static inline vs inline was decided when the Py_INCREF() macro was converted to a static inline function: python/cpython#79240 especially this comment: python/cpython#79240 (comment)

I don't want to diminish your work, but I don't see any "rationale" in this discussion, just a number of comments that seem to be talking past each other, and an arbitrary decision from you at the end :-)

So far, I didn't see any conclusive analysis that static inline functions are really faster than regular function calls, especially in third party C extensions.

I agree that performance claims should generally be backed with benchmark numbers, but I don't think a lot of analysis is required to conclude that Py_INCREF will generally be faster if inlined than if called through a regular function call.

Yes, most C extensions might not benefit from inlining of trivial functions such as Py_INCREF, PyTuple_GET_ITEM, etc. But some of them will. Even if they are a minority, you can't ignore that part of the ecosystem.

[vstinner]

I agree that performance claims should generally be backed with benchmark numbers, but I don't think a lot of analysis is required to conclude that Py_INCREF will generally be faster if inlined than if called through a regular function call. Yes, most C extensions might not benefit from inlining of trivial functions such as Py_INCREF, PyTuple_GET_ITEM, etc.

This issue isn't about Py_INCREF() or PyTuple_GET_ITEM(), but more general. I'm not suggesting to convert Py_INCREF() or PyTuple_GET_ITEM() to regular functions.

I'm just saying that we should design a guideline to decide which functions are seen as performance critical and should have a faster variant, and which ones don't need that. I don't think that all of the ~1600 C API functions need to have a faster variant (only a small numbers have it right now).

I don't want to diminish your work, but

I don't see the point of your remark unless you want to diminish my work. Maybe just omit it next time.

I don't see any "rationale" in this discussion, just a number of comments that seem to be talking past each other, and an arbitrary decision from you at the end :-)

I'm just giving pointers to past discussions if some people want to understand how static inline was selected. You see the choice as "arbitrary". Things changed since 2018, the C API has way more static inline functions, constraints about performance changes, we have a better understand on how the C API is consumed, etc.

We can move from static inline to something else if needed. I don't have a strong opinion about it. I just saw that so far, we didn't get major issues specific to static inline. I just vaguely recall issues with inline and duplicate function definitions in different C files. If someone needs more details, I can try to look into the details (that I now forgot).

[vstinner]

Message moved to a new issue: Define a public "maximize performance" API and separate it better from the regular "maximize compatibility" API.

[pitrou]

I'm just saying that we should design a guideline to decide which functions are seen as performance critical and should have a faster variant, and which ones don't need that.

This is derailing the discussion from the subject of this issue ("avoid macros and static inline functions"). Perhaps you should open a separate issue for that guideline?