Skip to content

Commit e362481

Browse files
committed
chg: [i2p crawler] filter I2P Website Unreachable and Website Unknown - Website Not Found in Addressbook domains
1 parent 180ca15 commit e362481

File tree

1 file changed

+0
-5
lines changed

1 file changed

+0
-5
lines changed

bin/crawlers/Crawler.py

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -447,11 +447,6 @@ def save_capture_response(self, parent_id, entries):
447447
dom_hash.add(self.date.replace('/', ''), item)
448448
dom_hash.add_correlation('domain', '', self.domain.id)
449449

450-
if self.domain.id.endswith('.i2p'):
451-
if
452-
if title_content == 'Website Unknown' or title_content == 'Website Unreachable':
453-
return False
454-
455450
gzip64encoded = crawlers.get_gzipped_b64_item(item.id, entries['html'])
456451
# send item to Global
457452
relay_message = f'crawler {gzip64encoded}'

0 commit comments

Comments
 (0)