行业研报这个数据处理跑失败了

[zanguixuan3]

https://new.opencsg-stg.com/datapipelines/dataflowInfo?id=127&type=pipeline&jobType=data_refine 我在 stg 也放了数据，可以拿这个数据集测试！初步判断是original_time 这个字段有的文件没值，有的文件有值，导致的解析失败。

[CheinTian]

自部署环境下报错详情：

[2025-12-24 07:15:48,760: INFO/MainProcess] Saved file to '/data/dataflow/test_93a1cb2a-64d9-4973-8686-6670c760d733/input/test.jsonl'
Downloading test2.jsonl:   0%|          | 0.00/26.0k [00:00<?, ?B/s]
Downloading test2.jsonl: 100%|##########| 26.0k/26.0k [00:00<00:00, 61.9MB/s]
[2025-12-24 07:15:48,855: INFO/MainProcess] Saved file to '/data/dataflow/test_93a1cb2a-64d9-4973-8686-6670c760d733/input/test2.jsonl'
2025-12-24 07:15:48.859 | INFO     | data_engine.ingester.csghub_ingester:ingest:54 - result: /data/dataflow/test_93a1cb2a-64d9-4973-8686-6670c760d733/input, _src_path: /data/dataflow/test_93a1cb2a-64d9-4973-8686-6670c760d733/input
2025-12-24 07:15:48.859 | INFO     | data_engine.core.executor:run:217 - Data ingested from /data/dataflow/test_93a1cb2a-64d9-4973-8686-6670c760d733/input
2025-12-24 07:15:48.904 | INFO     | data_engine.core.executor:run:233 - Setting up data formatter...
2025-12-24 07:15:48.942 | INFO     | data_engine.utils.file_utils:find_files_with_suffix:78 - params suffixes defined: ['.jsonl', '.json', '.parquet', '.csv', '.txt', '.tsv', '.jsonl.zst']
2025-12-24 07:15:48.950 | INFO     | data_engine.utils.file_utils:find_files_with_suffix:99 - handling files: {'.jsonl': ['/data/dataflow/test_93a1cb2a-64d9-4973-8686-6670c760d733/input/test2.jsonl', '/data/dataflow/test_93a1cb2a-64d9-4973-8686-6670c760d733/input/test.jsonl']}
2025-12-24 07:15:48.950 | INFO     | data_engine.core.executor:run:250 - Loading dataset from data formatter...
2025-12-24 07:15:48.982 | INFO     | data_engine.utils.file_utils:find_files_with_suffix:78 - params suffixes defined: ['.jsonl']
2025-12-24 07:15:48.984 | INFO     | data_engine.utils.file_utils:find_files_with_suffix:99 - handling files: {'.jsonl': ['/data/dataflow/test_93a1cb2a-64d9-4973-8686-6670c760d733/input/test2.jsonl', '/data/dataflow/test_93a1cb2a-64d9-4973-8686-6670c760d733/input/test.jsonl']}
Generating jsonl split: 0 examples [00:00, ? examples/s]
Generating jsonl split: 4 examples [00:00, 289.47 examples/s]
[2025-12-24 07:15:50,062: WARNING/MainProcess] Traceback (most recent call last):
[2025-12-24 07:15:50,062: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 1989, in _prepare_split_single
    writer.write_table(table)
[2025-12-24 07:15:50,062: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/arrow_writer.py", line 584, in write_table
    pa_table = table_cast(pa_table, self._schema)
[2025-12-24 07:15:50,062: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/table.py", line 2240, in table_cast
    return cast_table_to_schema(table, schema)
[2025-12-24 07:15:50,063: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/table.py", line 2199, in cast_table_to_schema
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
[2025-12-24 07:15:50,063: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/table.py", line 2199, in <listcomp>
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
[2025-12-24 07:15:50,063: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/table.py", line 1793, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
[2025-12-24 07:15:50,063: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/table.py", line 1793, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
[2025-12-24 07:15:50,063: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/table.py", line 2065, in cast_array_to_feature
    return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
[2025-12-24 07:15:50,063: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/table.py", line 1795, in wrapper
    return func(array, *args, **kwargs)
[2025-12-24 07:15:50,063: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/table.py", line 1937, in array_cast
    return array.cast(pa_type)
[2025-12-24 07:15:50,063: WARNING/MainProcess]   File "pyarrow/array.pxi", line 935, in pyarrow.lib.Array.cast
[2025-12-24 07:15:50,063: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/pyarrow/compute.py", line 400, in cast
    return call_function("cast", [arr], options, memory_pool)
[2025-12-24 07:15:50,063: WARNING/MainProcess]   File "pyarrow/_compute.pyx", line 572, in pyarrow._compute.call_function
[2025-12-24 07:15:50,064: WARNING/MainProcess]   File "pyarrow/_compute.pyx", line 367, in pyarrow._compute.Function.call
[2025-12-24 07:15:50,064: WARNING/MainProcess]   File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
[2025-12-24 07:15:50,064: WARNING/MainProcess]   File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
[2025-12-24 07:15:50,064: WARNING/MainProcess] pyarrow.lib.ArrowInvalid: Failed to parse string: '' as a scalar of type timestamp[s]
[2025-12-24 07:15:50,064: WARNING/MainProcess] 
The above exception was the direct cause of the following exception:
[2025-12-24 07:15:50,064: WARNING/MainProcess] Traceback (most recent call last):
[2025-12-24 07:15:50,064: WARNING/MainProcess]   File "/dataflow/data_celery/job/tasks.py", line 78, in run_pipline_job
    run_pipline_job_task(recipe_model, job_obj, db_session, user_id, user_name, user_token)
[2025-12-24 07:15:50,064: WARNING/MainProcess]   File "/dataflow/data_celery/job/tasks.py", line 206, in run_pipline_job_task
    _, branch_name = executor.run()
[2025-12-24 07:15:50,065: WARNING/MainProcess]   File "/dataflow/data_engine/core/executor.py", line 255, in run
    dataset = self.formatter.load_dataset(load_data_np, self.cfg)
[2025-12-24 07:15:50,065: WARNING/MainProcess]   File "/dataflow/data_engine/format/mixture_formatter.py", line 135, in load_dataset
    dataset = formatter.load_dataset(num_proc, global_cfg)
[2025-12-24 07:15:50,065: WARNING/MainProcess]   File "/dataflow/data_engine/format/formatter.py", line 65, in load_dataset
    datasets = load_dataset(self.type,
[2025-12-24 07:15:50,065: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/load.py", line 2582, in load_dataset
    builder_instance.download_and_prepare(
[2025-12-24 07:15:50,065: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 1005, in download_and_prepare
    self._download_and_prepare(
[2025-12-24 07:15:50,065: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 1100, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
[2025-12-24 07:15:50,065: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 1860, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
[2025-12-24 07:15:50,065: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 2016, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
[2025-12-24 07:15:50,065: WARNING/MainProcess] datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
[2025-12-24 07:15:50,133: INFO/MainProcess] Task run_pipline_job[22d50f31-3854-40c9-9c97-eb2b37df2003] succeeded in 14.267213441431522s: False

[CheinTian]

prod: https://opencsg-stg.com/datasets/CheinTian/test-failed?tab=summary
stg: https://opencsg.com/datasets/chl.tian/test-failed?tab=files&actionName=files

stg和 prod 环境也已复现，上面是用到的数据集。

[HaiHui886]

fixed in #131