Skip to content

[IO] Add Redis Connector#289

Open
gypleon wants to merge 7 commits intohusky-team:experimentalfrom
gypleon:dev
Open

[IO] Add Redis Connector#289
gypleon wants to merge 7 commits intohusky-team:experimentalfrom
gypleon:dev

Conversation

@gypleon
Copy link

@gypleon gypleon commented Jun 26, 2017

Test cases

- Preparation

  1. Flush Redis data using "/data/opt/course/redis/flushall-cluster.sh"
  2. Check configuration file "$HUSKY_ROOT/test/test-redis-config.ini"

- Test 1: CMake

  1. Follow the standard Cmake commands of Husky.
  2. make "Master".
  3. make "TestRedisOutputFormat".
  4. make "TestRedisInputOutputFormat".
  5. make "WordCountRedis".

Expectation: running well.
Notes:

- Test 2: enwiki from MongoDB to Redis:

  1. Run "./Master -C test-redis-config.ini"
  2. Run "time /data/opt/tmp/exec.sh ./TestRedisOutputFormat -C test-redis-config.ini"

Expectation: running well. 4825150 records to be imported. around 1m30s.
Notes: MUST wait for 10 minutes before Test 3, because for the first time data were loaded into Redis cluster, it takes time for master-slave synchronization.

- Test 3: enwiki from Redis to Redis:

  1. Run "./Master -C test-redis-config.ini"
  2. Run "time /data/opt/tmp/exec.sh ./TestRedisInputOutputFormat -C test-redis-config.ini"

Expectation: running well. 4825150 records to be imported. around 1m.
Notes: On Husky-worker side, "waiting for keys" logs may be printed as expected, which means that some workers are waiting for keys from the next batch.

- Test 4: enwiki WordCount:

  1. Run "./Master -C test-redis-config.ini"
  2. Run "time /data/opt/tmp/exec.sh ./WordCountRedis -C test-redis-config.ini"

Expectation: running well. 4825150 records to be imported. around 50s.
Notes: Can be performed consecutively without restart Husky-Master.

@gypleon
Copy link
Author

gypleon commented Jun 26, 2017

OMG!!! I merged these commits for several hours... but they come out again... maybe I made a ridiculous mistake that override the merge operations.... OMG.... actually, "[IO] Add Redis Connector. Fixed merge conflicts." (38ce3c2) is supposed to be the ONLY ONE left.

I am currently searching for safely merging solutions.

@gypleon gypleon force-pushed the dev branch 3 times, most recently from f322ea9 to 95228f4 Compare June 27, 2017 17:06
@gypleon
Copy link
Author

gypleon commented Jun 27, 2017

Hi Legend,
I think the commit problem has been fixed. Please review and leave me comments.
Thanks.

@kygx-legend
Copy link
Member

This PR is including other commits in master branch which are not merged into experimental branch. Could you redo the rebase?

@gypleon
Copy link
Author

gypleon commented Jun 28, 2017

No problem.

@kygx-legend kygx-legend changed the title Add Redis Connector. Add Redis Connector Jun 28, 2017
@kygx-legend kygx-legend changed the title Add Redis Connector [IO] Add Redis Connector Jun 28, 2017
@kygx-legend
Copy link
Member

This branch is updated to master now. Please rebase again!

@kygx-legend
Copy link
Member

And you can run ./scripts/clang-format.py -o check to see what does not match the coding style and run ./scripts/clang-format.py -o replace to fix those errors automatically. Then use lint.py to double check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments