|
1 | 1 | # OpenObserve: Introduction |
2 | 2 |
|
3 | | -`OpenObserve` is a cloud native observability platform (`Logs`, `Metrics`, `Traces`) that provides `~140x lower storage costs` (compared to Elasticsearch. YMMV. Could be higher or lower based on data entropy) for real life log data, significantly lower operational cost and ease of use. It can scale to petabytes of data, is highly performant and allows you to sleep better at night 😀. If you are looking for an observability tool for logs, metrics and traces, do evaluate OpenObserve and how its approach towards observability could help you build better software, save money on observability costs, and sleep better. |
4 | | - |
5 | | - |
6 | | -## Project Status, Features and Roadmap |
7 | | - |
8 | | -Following is the list of available features and roadmap. |
9 | | - |
10 | | -| # | Feature | Status | |
11 | | -|---|---------------------------------------------------------------|---------------------| |
12 | | -| 1 | Log search | Available | |
13 | | -| 2 | Highly compressed storage of data | Available | |
14 | | -| 3 | Dynamic evolution of schema | Available | |
15 | | -| 4 | Out of the box authentication | Available | |
16 | | -| 5 | Support of S3, MinIO, GCS, Azure blob for data storage | Available | |
17 | | -| 6 | Advanced GUI | Available | |
18 | | -| 7 | SQL based query language | Available | |
19 | | -| 8 | Support for very high cardinality data | Available | |
20 | | -| 9 | Search-around logs data | Available | |
21 | | -| 10 | User defined Ingest and Query functions (VRL based) | Available | |
22 | | -| 11 | Multi-tenancy | Available | |
23 | | -| 12 | Ingestion API compatibility with Elasticsearch | Available | |
24 | | -| 13 | Search and aggregation API compatibility with Elasticsearch | [Through zPlane](zplane) | |
25 | | -| 14 | Standard alerts (Based on logs) | Available | |
26 | | -| 15 | Real time Alerts (Based on logs) | Available | |
27 | | -| 16 | High Availability (HA) and clustering | Available | |
28 | | -| 17 | Stateless nodes | Available | |
29 | | -| 18 | Localization for multiple languages | Available | |
30 | | -| 19 | Prebuilt binaries for multiple platforms | Available | |
31 | | -| 20 | Prebuilt container images for multiple platforms | Available | |
32 | | -| 21 | Prebuilt container images for with SIMD acceleration | Available | |
33 | | -| 22 | SIMD support for vectorized processing (AVX512 and Neon) | Available | |
34 | | -| 23 | Dashboards | Available | |
35 | | -| 24 | Metrics | Available | |
36 | | -| 25 | PromQL support for metrics | Available (97% PromQL compliant) | |
37 | | -| 26 | Traces | Available | |
38 | | -| 27 | Standard alerts (Based on metrics) | Available | |
39 | | -| 28 | Real time Alerts (Based on metrics) | Available | |
40 | | -| 29 | Template based alert target (Allows alerting to slack, teams and many more) | Available | |
41 | | -| 30 | Send alerts to Prometheus alertmanager | Available | |
42 | | -| 31 | Ingest AWS logs (cloudwatch, VPC flow logs, AWS WAF and more) using Kinesis firehose | Available | |
43 | | -| 32 | Single Sign On(SSO) | Available (Enterprise) | |
44 | | -| 33 | RBAC (Role Based Access Control) | Available (Enterprise) | |
45 | | -| 34 | Front end - Performance analytics | Available | |
46 | | -| 35 | Front end - Session Replay | Available | |
47 | | -| 36 | Front end - Error tracking | Available | |
48 | | -| 37 | Log patterns | To start | |
49 | | -| 38 | Anomaly detection | To start | |
50 | | -| 39 | Correlation between logs, metrics and traces | To start | |
51 | | -| 40 | Dashboard migration from Splunk, Kibana and Grafana | beta. [https://dc.openobserve.ai](https://dc.openobserve.ai) [https://github.com/openobserve/dashboard_converter](https://github.com/openobserve/dashboard_converter) | |
| 3 | +## What is OpenObserve? |
52 | 4 |
|
| 5 | +**OpenObserve**, also referred to as O2, is a cloud native observability platform that unifies **logs, metrics, and traces** into a single, powerful solution. Built from the ground up for modern cloud environments, OpenObserve delivers enterprise-grade observability at a fraction of the cost and complexity of traditional solutions. |
53 | 6 |
|
54 | | -Please raise any new feature requests via [github issue tracker](https://github.com/openobserve/openobserve/issues). |
| 7 | +<iframe width="560" height="315" src="https://www.youtube.com/embed/C0GH5Ox8NnY?si=Xm-tEpbnm-iywGqH" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe> |
55 | 8 |
|
56 | | -You can use either the open source version or [OpenObserve Cloud](https://cloud.openobserve.ai). [OpenObserve Cloud](https://cloud.openobserve.ai) is built on top of open source OpenObserve but has minor differences to account for its SaaS nature. We will highlight the differences in the documentation whenever needed. |
| 9 | +## Why Choose OpenObserve? |
57 | 10 |
|
58 | | -## Guiding principles |
| 11 | +### Dramatic Cost Reduction |
| 12 | +Experience up to **140x lower storage costs** compared to Elasticsearch while maintaining full functionality. Our innovative architecture ensures you get more observability for less budget. |
59 | 13 |
|
60 | | -We want to build the best software in the observability category in the world, and we believe that the below principles will keep us aligned towards that: |
| 14 | +### Effortless Scale |
| 15 | +Built to handle **petabyte-scale data** with ease. Whether you're a growing startup or a large enterprise, OpenObserve scales seamlessly with your needs without the operational overhead. |
61 | 16 |
|
62 | | -1. Day 1: It should be easy to setup and use |
63 | | - 1. You should be able to install (for self hosted option) or sign up (for SaaS platform) in under 2 minutes. |
64 | | - 1. You should be able to start ingesting data in under 2 minutes and start observing the behavior of your applications without any major configuration. |
65 | | -2. Day 2: It should not be painful to keep the system up and running |
66 | | - 1. Application should be stable and in the case of issues should be able to heal itself automatically. |
67 | | - 1. Majority of the users should be able to start using the system efficiently with ZERO configuration. |
68 | | - 1. Scaling up/down should be as easy as changing the number of nodes in an autoscaling group (in AWS) or changing the number of replicas (in k8s). |
69 | | - 1. Majority of the folks should not need backups or should be able to do it without DBA level skills. |
70 | | - 1. Fear of upgrades should not make you lose your sleep |
71 | | -3. Features and Usability: It should have good features and functionality to do the job efficiently |
72 | | - 1. System should be highly usable from the get go - providing excellent ROI on the invested time. A great UI and API are important to achieve it. |
73 | | - 1. Logs themselves do not provide you visibility into your application. You need metrics and traces as well and the ability to correlate them. |
74 | | -4. Cost: It should be cost effective |
75 | | - 1. You should not have to mortgage your house or company assets in order to run the system either in self hosted mode (with or without licensing cost) or for SaaS platform. |
76 | | -5. Learning curve: It should allow beginners to do a lot of tasks easily and advanced users should be able to use most of their existing skills |
77 | | - 1. A user who has never used the system should be able to set up and use the system efficiently for basic needs or should be able to use existing skills for advanced purposes. |
78 | | -6. Performance: It should be highly performant |
79 | | - 1. System should be highly performant for most of the use cases in the real world. |
80 | | - 1. Many a times performance requires a tradeoff. In situations of tradeoffs, it should be generally acceptable to the majority of the users for the use case with excellent tradeoff value in return. |
| 17 | +### Performance First |
| 18 | +Engineered for speed with SIMD acceleration and vectorized processing. Get faster queries, real-time insights, and better performance across all your observability data. |
81 | 19 |
|
82 | | -## How does OpenObserve compare to Elasticsearch |
| 20 | +### Unified Experience |
| 21 | +Stop juggling multiple tools. OpenObserve brings logs, metrics, traces, frontend monitoring, and alerting into one cohesive platform with a single pane of glass. |
83 | 22 |
|
84 | | -Elasticsearch is a general purpose search engine which can be used for app search or log search. OpenObserve is built specifically for log search. If you are looking for a lightweight alternative to Elasticsearch then you should take a look at ZincSearch. |
| 23 | +## Who Should Use OpenObserve? |
85 | 24 |
|
86 | | -OpenObserve provides ability to index data in multiple ways to make it faster yet keep storage size low. It uses a combination of partitioning, bloom filters, inverted indexes, caching and columnar storage to make search and aggregation queries faster. You can combine these to find the right balance between storage and performance. [Uber found 80% of queries in their production environment to be aggregation queries](https://www.uber.com/en-IN/blog/logging/) and columnar data storage of OpenObserve means that aggregation queries will typically be much faster than Elasticsearch. |
| 25 | +OpenObserve is perfect for: |
87 | 26 |
|
88 | | -Below is the result when we sent real life log data from our kubernetes cluster to both Elasticsearch and OpenObserve using fluentbit. This only pertains to storage. Cost of EBS volume is [8 cents/GB/Month (GP3)](https://aws.amazon.com/ebs/pricing/), cost of s3 is [2.3 cents/GB/month](https://aws.amazon.com/s3/pricing/). In HA mode in Elasticsearch you generally have 1 primary node and 2 replicas. You don't need to replicate s3 for data durability/availability as [AWS redundantly stores your objects on multiple devices across a minimum of three Availability Zones (AZs) in an Amazon S3 Region](https://aws.amazon.com/s3/faqs/). |
| 27 | +- **Engineering teams** looking to reduce observability costs without sacrificing capabilities |
| 28 | +- **DevOps professionals** who need reliable, scalable monitoring at any scale |
| 29 | +- **Organizations** migrating from expensive legacy solutions like Elasticsearch |
| 30 | +- **Companies** requiring comprehensive observability across cloud-native applications |
89 | 31 |
|
90 | | - |
| 32 | +## Key Advantages |
91 | 33 |
|
92 | | -OpenObserve offers significant advantage of 140x lower storage costs compared to Elasticsearch in the above scenario (YMMV, you could get higher or lower values based on entropy of data). That does not even consider additional unused EBS volume capacity that needs to be available in order to not run out of disk space and the effort that it requires to keep monitoring disk usage so it is not filled. |
| 34 | +- **Cost Effective**: Dramatically lower storage and operational costs |
| 35 | +- **Cloud Native**: Built for modern cloud environments and containerized workloads |
| 36 | +- **Easy to Deploy**: Get started quickly with minimal configuration |
| 37 | +- **Highly Compatible**: Works with existing Prometheus, Elasticsearch tooling and workflows |
| 38 | +- **Enterprise Ready**: SSO, RBAC, and compliance features available |
93 | 39 |
|
94 | | -Stateless node architecture allows OpenObserve to scale horizontally without worrying about data replication or corruption challenges. |
| 40 | +## Ready to Get Started? |
95 | 41 |
|
96 | | -OpenObserve's lack of index mapping and associated challenges provides a hassle-free experience in managing clusters. |
97 | | - |
98 | | -You will typically see much lower operational effort and cost in managing OpenObserve clusters compared to Elasticsearch. |
99 | | - |
100 | | -The platform's built-in GUI eliminates the need for another component like Kibana, and has awesome performance, thanks to Rust, without the challenges of JVM. |
101 | | - |
102 | | -In contrast to Elasticsearch, which is a general-purpose search engine that doubles as an observability tool, OpenObserve was built from the ground up as an observability tool, with high focus on delivering exceptional observability. |
103 | | - |
104 | | -## Elasticsearch compatibility |
105 | | - |
106 | | -OpenObserve `_bulk` API endpoint is elasticsearch compatible and can be used by log forwarders like fluentbit, fluentd and vector. Filebeat is supported through zPlane. |
107 | | - |
108 | | -Search and aggregation API compatibility with Elasticsearch is provided through zPlane. |
109 | | - |
110 | | -zPlane is the enterprise product offered by ZincLabs that among other things provides Elasticsearch search and aggregation compatibility. Learn more about it at [zPlane docs](zplane) |
111 | | -## Are there any benchmarks? |
112 | | -OpenObserve is currently under heavy development with many changes still happening to the core engine. We will do benchmarking soon as we complete implementation of some of the items at hand. |
113 | | - |
114 | | -In the meanwhile, there are hundreds of production installations of OpenObserve globally at small, mid tier and very large scale being used by startups and enterprises alike. Many have reported that OpenObserve is highly performant. Some of them have replaced 5-7 node Elasticsearch clusters with a single node of OpenObserve. |
115 | | - |
116 | | -Here is a [case study of Jidu](https://openobserve.ai/blog/jidu-journey-to-100-tracing-fidelity) that increased their throughput and query performance by 10x and reduced their storage costs by 10x by switching from Elasticsearch to OpenObserve, ingesting 10TB of data everyday. Jidu is a large EV manufacturer in China. |
| 42 | +OpenObserve's architectural approach can transform how you handle observability - reducing costs while improving performance and ease of use. |
117 | 43 |
|
| 44 | +**Next Steps:** |
118 | 45 |
|
| 46 | +- Explore our comprehensive [Feature List](../features/logs.md) to see all capabilities |
| 47 | +- Check out [Getting Started Guide](../getting-started.md) to start exploring |
| 48 | +- Join our [Community](https://github.com/openobserve/openobserve/discussions) to connect with other users |
119 | 49 |
|
| 50 | +*Sleep better at night knowing your observability stack is both powerful and affordable* |
0 commit comments