I have done some workaround across implementing ActiveMq to the messagebus so would like to contribute #350
Conversation
hdr and redir persistence + cityhash64 + graceful shutdown fix + kafka SSL
imho, in hbase queue's `get_next_requests` method, params: min_hosts and max_requests_per_host are optional, because this method will check whether they are None, eg line no.225 and no.236, but kwargs.pop will raise Exception if I don't give these two params.
the `from_worker` method is duplicate , base class `BaseCrawlingStrategy` define `from_worker` already.
Do not remove existing loggers on configuration
Add message type logging and time for batch
Improving errors logging in async offsets fetcher
Smaller kafka client poll intervals
Update hbase.py
maybe miss b?
update BCPerHostLimit in cluster example
HBase states table option
Add Scrapy extension to export stats to Kafka Use mixin class to create new db/sw classes Extend messagebus class with stats logic Extend codec logic with encoding/decoding stats Use export-to-tsdb logic from bigbot-common package Use single logic for forming SSL params
Move happybase to dependencies folder Add get_stats() to distributed backend interface Provide useful context manager to measure elapsed time Use the manager to estimate get_stats() efficiency
|
@amitsing89 You should rebase your changes on top of the latest |
Solving the problem of 'Decode spider logs self.body return none error' #374
Update models.py
Bump to Python 3.6 + 3.7
add assertion error description for easier debugging
|
This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
|
This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation. |
12 participants
Hi,
I have started once to integrate the activeMq in the messagebus so i would like to add that to the repository, so if you can allow kme to contribute then it would be helpful to add a new component