Data Tools with Twitter Ingestion Server, Twitter GeoTagger and AsterixDB Ingestion Server.

[baiqiushi]

Data Tools

Data Tools is a new module consisting of 3 components that serve the data preparation of the TwitterMap application.

Twitter Ingestion Server

Twitter Ingestion Server is a daemon service that can ingest real-time tweets from Twitter Filter Stream API into local gzip files in a daily rotation manner.
It is also a light-weight HTTP server with 3 endpoints:

• /stats - HTTP GET endpoint that returns current ingestion status information in JSON format.
• /proxy - WebSocket endpoint that pushes real-time tweets to any client in connection.
• / - HTTP GET endpoint that returns an index.html as an example page demonstrating the usage of the above two endpoints.

Twitter GeoTagger

Twitter GeoTagger is Java program to geoTag Twitter JSON with {stateID, stateName, countyID, countyName, cityID, cityName}.
It has 2 modes,

• 1. in API mode, it provides a function tagOneTweet that can be called from other programs;
• 2. in process mode, it provides a main function that can be started as a JVM process to geotag tweets in shell console pipeline.

AsterixDB Ingestion Server

TBD.

[sadeemsaleh]

NOT ready to be merged

[codecov-io]

Codecov Report

Merging #807 (6bd3e50) into master (9caf3d0) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #807   +/-   ##
=======================================
  Coverage   63.91%   63.91%           
=======================================
  Files          75       75           
  Lines        4076     4076           
  Branches      355      355           
=======================================
  Hits         2605     2605           
  Misses       1471     1471           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9caf3d0...6bd3e50. Read the comment docs.

[codecov-commenter]

Codecov Report

Merging #807 (0538333) into master (9caf3d0) will not change coverage.
The diff coverage is n/a.

❗ Current head 0538333 differs from pull request most recent head 3786af4. Consider uploading reports for the commit 3786af4 to get more accurate results

@@           Coverage Diff           @@
##           master     #807   +/-   ##
=======================================
  Coverage   63.91%   63.91%           
=======================================
  Files          75       75           
  Lines        4076     4076           
  Branches      355      355           
=======================================
  Hits         2605     2605           
  Misses       1471     1471           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9caf3d0...3786af4. Read the comment docs.