Refactor Python UDF Execution to In-Process

[wudidapaopao]

chDB currently supports Python user-defined functions (UDFs) in queries, leveraging ClickHouse's native capability.
The existing implementation uses standard input/output as the communication channel and executes UDFs in a separate Python process.

While this approach is functional, it is not the most elegant or flexible solution for chDB. We propose refactoring the UDF execution mechanism to run UDFs directly within the main chDB process.

Key Benefits of In-Process UDF Execution:

1. Superior Performance: Remove overhead from inter-process communication, enabling faster UDF execution. We can also facilitate better optimization for batch processing scenarios.

2. Extended Flexibility: Lay the groundwork for supporting more advanced UDF types in the future, such as custom aggregate functions and custom table functions.

import chdb
from chdb.udf import chdb_udf

@chdb_udf()
def sum_udf(lhs, rhs):
    return int(lhs) + int(rhs)

chdb.query("select sum_udf(12, 22)").show()

[cyrusmsk]

Some details of how this could be implemented would be interesting.

[auxten]

Some details of how this could be implemented would be interesting.

It will be an in-process version similar to the impl of Python Table Function.
No Ser/Der needed to call it. CC @sichenzhao

[cyrusmsk]

Could you put a link here for Python table function?
Implementation code

[niyue]

refactoring the UDF execution mechanism to run UDFs directly within the main chDB process

I’m not fully sure how multi-threading would impact performance in this setup. Will we consider leveraging pybind11’s Python 3.12+ sub-interpreter support [1], so that UDFs can be executed in parallel across multiple interpreters, allowing better utilization of multi-core CPUs?

[1] https://pybind11.readthedocs.io/en/stable/advanced/misc.html#sub-interpreter-support