Possible reason 1: when you use from pyspark.SQL.Functions import * to pour in the pyspark function, the python built-in function in UDF is replaced by spark function, and you can import it again
Possible reason 2: the user-defined UDF function is not placed in the main function, resulting in an error
IndexError: invalid index to scalar variable.
invalid index
generally appears in the np array index error
Here are some examples:
a = np.array([[1, 2], [3, 4], [5, 6]])
print(a[0][0][1])
After running, report an error:
IndexError: invalid index to scalar variable.
Therefore, when the above error occurs, we should check whether the index is wrong, such as the original two-dimensional array, using three levels of index.