Skip to content

fix: Replace synchronization error #1951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shaohuzhang1 merged 1 commit into from
Dec 30, 2024
Merged

fix: Replace synchronization error #1951

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/dataset/task/sync.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ def sync_web_dataset(dataset_id: str, url: str, selector: str):
max_kb_error.error(f'同步web知识库:{dataset_id}出现错误{str(e)}{traceback.format_exc()}')


@celery_app.task(base=QueueOnce, once={'keys': ['dataset_id']}, name='celery:sync_web_dataset')
@celery_app.task(base=QueueOnce, once={'keys': ['dataset_id']}, name='celery:sync_replace_web_dataset')
def sync_replace_web_dataset(dataset_id: str, url: str, selector: str):
try:
max_kb.info(f"开始--->开始同步web知识库:{dataset_id}")
Copy link
Contributor Author

@shaohuzhang1 shaohuzhang1 Dec 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The task sync_replace_web_dataset should have been called from the correct place or with an appropriate method. The function is defined in the file /data/xxx.py, where it seems like it's meant to replace existing data with new data when syncing web datasets.

Here are some general points:

  1. Task Name: The name of the task (celery:sync_replace_web_dataset) might be misleading since there could already be a celery:sync_web_dataset.
  2. Function Logic:
    • Ensure that the logic within this function aligns with your requirements, especially considering you're replacing data rather than adding more.
    • Make sure necessary checks are implemented (like validating URLs or selectors).
  3. Error Handling:
    • It would be beneficial to print error messages clearly and ensure that logs indicate what went wrong during execution.
  4. Testing:
    • Consider creating tests for this function if its behavior needs to be verified after code changes.
  5. Documentation:
    • Add comments and documentation explaining the purpose and usage of this function, ideally including examples if applicable.

To fix the current implementation, consider renaming the task and ensuring it fits into the overall flow of your codebase. If needed, refactor the function to better fit the replacement functionality instead of just logging errors upon failure.

Expand Down
Loading