pyspark.sql.tvf.TableValuedFunction.python_worker_logs

TableValuedFunction.python_worker_logs()

Returns a DataFrame of logs collected from Python workers.

New in version 4.1.0.

Returns

DataFrame

Examples

>>> import pyspark.sql.functions as sf
>>> import logging
>>>
>>> @sf.udf("string")
... def my_udf(x):
...     logger = logging.getLogger("my_custom_logger")
...     logger.warning("This is a warning")
...     return str(x)
...
>>> spark.conf.set("spark.sql.pyspark.worker.logging.enabled", "true")
>>> spark.range(1).select(my_udf("id")).show()
+----------+
|my_udf(id)|
+----------+
|         0|
+----------+
>>> spark.tvf.python_worker_logs().select(
...     "level", "msg", "context", "logger"
... ).show(truncate=False)  
+-------+-----------------+---------------------+----------------+
|level  |msg              |context              |logger          |
+-------+-----------------+---------------------+----------------+
|WARNING|This is a warning|{func_name -> my_udf}|my_custom_logger|
+-------+-----------------+---------------------+----------------+