Few question regarding hugegraph

[mkj-git]

I recently got to know about Hugegraph and i am very impressed with its features like API, multi DB support, backup & rollback, AI support. But one thing that surprise me why this isnt popular? I saw one discussion where few company has posted that they use this DB. My question is

1. Does this db really support 100B data as mentioned on website?
2. What is the latest future plan?
3. Is there any recent real world deployment of this DB in production?

[imbajin]

I recently got to know about Hugegraph and i am very impressed with its features like API, multi DB support, backup & rollback, AI support. But one thing that surprise me why this isnt popular? I saw one discussion where few company has posted that they use this DB. My question is

1. Does this db really support 100B data as mentioned on website?
2. What is the latest future plan?
3. Is there any recent real world deployment of this DB in production?

Thank you for your attention. Some context for it:

You can understand it as a new version of JG that was originally designed as a data structure rewrite and comprehensive enhancement. Later, it gradually evolved into its own independently managed RocksDB+Raft PD/storage system, and made significant improvements in many engineering areas. (Include Graph Computing/Toolchain/AI.. etc, not only a graph-db:)

You may wonder, since it has undergone so many optimizations and improvements and provided a relatively complete surrounding
ecosystem, why isn't it popular enough?

The main issue is that it lacks community operations people/skills, The field of graphics is relatively niche and has a high threshold. Even if you can provide better engineering practices and designs than Neo4j and JG, you still need a lot of energy to make ordinary people believe in it, and a lot of energy to do productization/documentation/blog related work.(Also lacks various official blogs, source code analysis, best practices, and more)

So indeed, the community lacks devs who are willing to participate and further promote it, whether it is in terms of code or documentation. We are also willing to guide everyone in our personal time

1. Does this db really support 100B data as mentioned on website?

Of course, if you know about HBase and Cassandra, they can support more than 100B of data stably. It's just a matter of optimizing read and write performance. Simply storing data is not difficult, the challenge is to read them efficiently, which is why we later introduced a distributed version of our home-grown Raft + RocksDB that no longer relies on external storage.

2. What is the latest future plan?

refer #2724 & https://github.com/apache/incubator-hugegraph/milestones

The other one is graduating from the Apache Incubator

3. Is there any recent real world deployment of this DB in production?

Yes, most of our existing PMC/Committers members are actual users (others parts in the user-case issue)

[mkj-git]

How does it perform in case of failure:

• if master node fails how re balancing happening

• Is this manual effort or handled by Hugegraph itself

• In case of Cassandra or Hbase, is hugegraph auto scaled supported?

[imbajin]

For the legacy version (<1.5),

changes to the storage layer are handled entirely by Hbase/Cassandra itself, and the graph side only needs to deal with the graph server, which is a nearly stateless service, so you can start multiple (e.g. in Docker/K8s) to implement a HA mechanism easily. (So in this version auto scaling is neither required nor implemented)

For the version ≥ 1.5 & if u use hstore as the backend, it follows a mechanism similar to tikv,except for node scaling, it is all handled by HG itself

[mkj-git]

In case of ≥ 1.5, if I use casssandra as back end, how a configuration will look like, is there any scalable cassandra backed configuration available?
In production which is recommended back end storage for scaled HG, Cassandra/Hbase/Hstore ?

[imbajin]

In case of ≥ 1.5, if I use casssandra as back end, how a configuration will look like, is there any scalable cassandra backed configuration available? In production which is recommended back end storage for scaled HG, Cassandra/Hbase/Hstore ?

First, if you are using any backend other than 'HStore', version 1.5 will not be significantly different from previous versions. You can refer to the default Cassandra configuration in the previous documentation (performance optimization is a separate matter, but it doesn't need to be addressed too early).

For the community, maintaining multiple backends is clearly costly. We recommend prioritizing backends that use Binary serialization (such as HBase/RocksDB). In the future, the community will significantly streamline unnecessary backends to reduce the difficulty of user selection. The core is to maintain the Raft + RocksDB combination (which can be single-node or distributed, and manages its own storage without any external component dependencies, making performance optimization and adjustment much easier). Otherwise, you will face a very large overhead in customizing and optimizing the performance of different backends (e.g., MySQL/ScyllaDB/HBase/Cassandra...).

Version 1.5 is the last version in which we will maintain compatibility with all backends. Starting with the next version, 1.7, we will reduce the number of backends by 50%, and by version 2.0, we will only keep backends that use Binary serialization and remove the Backend abstraction layer (users who want to use a compatible version can continue to use 1.5).

[mkj-git]

So you are suggesting Raft + RocksDB is the best combination for scalability & fault tolerant going forward?

Sorry my knowledge on Raft & RockDB is limited.

What I understood, might be wrong, that Raft is set of algorithms and RockDB is an embedded DB. When I hear something like embedded first things comes in my mind is that is that scalable & distributed in nature?

We recommend prioritizing backends that use Binary serialization (such as HBase/RocksDB).

I think HBase will be a good choice in a distributed world, considering its scalability and fault tolerance in distributed world? RockDB I am not sure as its embedded DB server, might good for test/dev purpose.

One more question does Hugegraph has any commercial version as well?

[imbajin]

So you are suggesting Raft + RocksDB is the best combination for scalability & fault tolerant going forward?

Sorry my knowledge on Raft & RockDB is limited.

What I understood, might be wrong, that Raft is set of algorithms and RockDB is an embedded DB. When I hear something like embedded first things comes in my mind is that is that scalable & distributed in nature?

We recommend prioritizing backends that use Binary serialization (such as HBase/RocksDB).

I think HBase will be a good choice in a distributed world, considering its scalability and fault tolerance in distributed world? RockDB I am not sure as its embedded DB server, might good for test/dev purpose.

One more question does Hugegraph has any commercial version as well?

1. Could refer FB's article: https://engineering.fb.com/2023/05/16/data-infrastructure/mysql-raft-meta/ (Or TiKV --> Its documentation and article introduction are comprehensive and clear)
2. The HBase/HDFS system is very classic, and I have used/dev with it before. If you are familiar with it, you can choose it as the backend, but it also has its own limitations. In addition, as mentioned above, RocksDB is actually widely used in distributed databases(u can search for it if you are interested), and it is not just used in an embedded storage engine (such as SQLite)
3. Nope for now (commercial)

[mkj-git]

Of course, if you know about HBase and Cassandra, they can support more than 100B of data stably. It's just a matter of optimizing read and write performance. Simply storing data is not difficult, the challenge is to read them efficiently, which is why we later introduced a distributed version of our home-grown Raft + RocksDB that no longer relies on external storage.

Is this tested on Hugegraph? what was the configuration setup?

Hugegraph looks very interesting to me because of all features it supports. But the problem is its all capabilties are not documented properly and that is what raising lots of confusion to me. Specially about its performance, scalability, fail over handling.

[imbajin]

Yep, both Hbase(2.x) and Cassandra(3.1x) have been tested, and if you're referring to the combination of Raft+RocksDB(HStore), there are also actual tests and configurations(Now it is also running in a production environment with billions of data). However, as you mentioned, there is a lack of documentation and specific instructions here, and we need time to improve and supplement it extensively, otherwise it will be unfamiliar to ordinary people.