feat: habilita/desabilita spider via API em vez de acesso direto ao banco - #1470
Open
diraol wants to merge 2 commits into
Open
feat: habilita/desabilita spider via API em vez de acesso direto ao banco#1470diraol wants to merge 2 commits into
diraol wants to merge 2 commits into
Conversation
…anco
O novo deployment (Kubernetes/Revoada) não expõe o PostgreSQL
publicamente. enable_spider/disable_spider no scheduler.py (usados
pelo workflow update_spider_status.yaml) agora chamam
PATCH /scraper/spiders/{spider_name} via QueridoDiarioAPIClient em vez
de abrir uma conexão SQLAlchemy direta. QUERIDODIARIO_DATABASE_URL
também deixou de ser enviada nos job settings da Zyte (_job_settings),
já que nenhum pipeline/extensão/monitor a utiliza mais.
Depende de okfn-brasil/querido-diario-api#108.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DdMDA5GSzUVdc2qTFSTEKH
caiotheodoro
left a comment
caiotheodoro
left a comment
There was a problem hiding this comment.
Small and clean. One risk inline on the new client method.
|
|
||
| PATCH /scraper/spiders/{spider_name} | ||
| """ | ||
| response = self.session.patch( |
There was a problem hiding this comment.
risk: this calls session.patch directly instead of going through a _patch helper like _get/_post. The session's Retry only allows GET/POST (line 41 above), so this PATCH gets none of the retry/backoff the rest of the client has - a transient 502/503 during the scheduled workflow fails hard instead of retrying.
Collaborator
Author
There was a problem hiding this comment.
Boa observação, obrigado. Corrigido em 5aeac9b: adicionei PATCH a allowed_methods do Retry (a operação é idempotente, só substitui o campo enabled) e extraí um helper _patch análogo a _get/_post para set_spider_enabled usar.
set_spider_enabled chamava session.patch diretamente, mas o Retry configurado na sessão só permite GET/POST, então uma falha transitória (502/503) durante o workflow agendado de enable/disable de spiders falhava sem retry. Adiciona PATCH a allowed_methods (a operação é idempotente, só substitui o campo enabled) e extrai um helper _patch análogo a _get/_post. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiEwVseUsoMM8ztc9iaz2Q
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
enable_spider/disable_spider(scheduler.py, usados pelo workflowupdate_spider_status.yaml) passam a chamarPATCH /scraper/spiders/{spider_name}viaQueridoDiarioAPIClient, em vez de abrir conexão SQLAlchemy direta com o Postgres.QUERIDODIARIO_DATABASE_URLremovida de_job_settings()— não é mais enviada como job setting pra Zyte, já queApiPipeline,StatsPersiste os monitors do Spidermon já usam a API desde a migração anterior.update_spider_status.yamlatualizado para usar os secretsQUERIDODIARIO_API_URL/QUERIDODIARIO_API_KEYem vez deQUERIDODIARIO_DATABASE_URL.Depende de okfn-brasil/querido-diario-api#108 (endpoint novo).
Test plan
pytest tests/test_scheduler.py— 6/6 passando (3 testes novos: enable/disable via client mockado,_job_settings()semQUERIDODIARIO_DATABASE_URL).QUERIDODIARIO_API_URL(feito em paralelo) e validarenable-spider/disable-spidermanualmente via workflow_dispatch.🤖 Generated with Claude Code