[SPARK-54820][PYTHON][4.0][4.1] Make pandas_on_spark_type compatible with numpy 2.4.0

backport #53580 to 4.0 and 4.1,
to restore https://github.com/apache/spark/actions/runs/20493084511/job/58888601618

Closes #53604 from zhengruifeng/bp_np_24.

Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>

Author: zhengruifeng

python/pyspark/pandas/typedef/typehints.py

@@ -342,7 +342,7 @@ def pandas_on_spark_type(tpe: Union[str, type, Dtype]) -> Tuple[Dtype, types.Dat
     try:
         dtype = pandas_dtype(tpe)
         spark_type = as_spark_type(dtype)
-    except TypeError:
+    except (TypeError, ValueError):
         spark_type = as_spark_type(tpe)
         dtype = spark_type_to_pandas_dtype(spark_type)
     return dtype, spark_type