Skip to content
This repository was archived by the owner on Nov 14, 2022. It is now read-only.

Commit 1fe9578

Browse files
zhixingheyi-tianzhixingheyi-tian
authored
Merge pull request #18 from HongW2019/doc-1.1.1
[PMEM-COMMON-17]Update documents for OAP 1.1.1
2 parents e1ce241 + 7d4bcd0 commit 1fe9578

File tree

4 files changed

+161
-23
lines changed

4 files changed

+161
-23
lines changed

‎CHANGELOG.md‎

Lines changed: 146 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,151 @@
11
# Change log
2-
Generated on 2021-04-29
2+
Generated on 2021-06-02
3+
4+
## Release 1.1.1
5+
6+
### Native SQL Engine
7+
8+
#### Features
9+
|||
10+
|:---|:---|
11+
|[#304](https://github.com/oap-project/native-sql-engine/issues/304)|Upgrade to Arrow 4.0.0|
12+
|[#285](https://github.com/oap-project/native-sql-engine/issues/285)|ColumnarWindow: Support Date/Timestamp input in MAX/MIN|
13+
|[#297](https://github.com/oap-project/native-sql-engine/issues/297)|Disable incremental compiler in CI|
14+
|[#245](https://github.com/oap-project/native-sql-engine/issues/245)|Support columnar rdd cache|
15+
|[#276](https://github.com/oap-project/native-sql-engine/issues/276)|Add option to switch Hadoop version|
16+
|[#274](https://github.com/oap-project/native-sql-engine/issues/274)|Comment to trigger tpc-h RAM test|
17+
|[#256](https://github.com/oap-project/native-sql-engine/issues/256)|CI: do not run ram report for each PR|
18+
19+
#### Bugs Fixed
20+
|||
21+
|:---|:---|
22+
|[#325](https://github.com/oap-project/native-sql-engine/issues/325)|java.util.ConcurrentModificationException: mutation occurred during iteration|
23+
|[#329](https://github.com/oap-project/native-sql-engine/issues/329)|numPartitions are not the same|
24+
|[#318](https://github.com/oap-project/native-sql-engine/issues/318)|fix Spark 311 on data source v2|
25+
|[#311](https://github.com/oap-project/native-sql-engine/issues/311)|Build reports errors|
26+
|[#302](https://github.com/oap-project/native-sql-engine/issues/302)|test on v2 failed due to an exception|
27+
|[#257](https://github.com/oap-project/native-sql-engine/issues/257)|different version of slf4j-log4j|
28+
|[#293](https://github.com/oap-project/native-sql-engine/issues/293)|Fix BHJ loss if key = 0|
29+
|[#248](https://github.com/oap-project/native-sql-engine/issues/248)|arrow dependency must put after arrow installation|
30+
31+
#### PRs
32+
|||
33+
|:---|:---|
34+
|[#332](https://github.com/oap-project/native-sql-engine/pull/332)|[NSE-325] fix incremental compile issue with 4.5.x scala-maven-plugin|
35+
|[#335](https://github.com/oap-project/native-sql-engine/pull/335)|[NSE-329] fix out partitioning in BHJ and SHJ|
36+
|[#328](https://github.com/oap-project/native-sql-engine/pull/328)|[NSE-318]check schema before reuse exchange|
37+
|[#307](https://github.com/oap-project/native-sql-engine/pull/307)|[NSE-304] Upgrade to Arrow 4.0.0|
38+
|[#312](https://github.com/oap-project/native-sql-engine/pull/312)|[NSE-311] Build reports errors|
39+
|[#272](https://github.com/oap-project/native-sql-engine/pull/272)|[NSE-273] support spark311|
40+
|[#303](https://github.com/oap-project/native-sql-engine/pull/303)|[NSE-302] fix v2 test|
41+
|[#306](https://github.com/oap-project/native-sql-engine/pull/306)|[NSE-304] Upgrade to Arrow 4.0.0: Change basic GHA TPC-H test target …|
42+
|[#286](https://github.com/oap-project/native-sql-engine/pull/286)|[NSE-285] ColumnarWindow: Support Date input in MAX/MIN|
43+
|[#298](https://github.com/oap-project/native-sql-engine/pull/298)|[NSE-297] Disable incremental compiler in GHA CI|
44+
|[#291](https://github.com/oap-project/native-sql-engine/pull/291)|[NSE-257] fix multiple slf4j bindings|
45+
|[#294](https://github.com/oap-project/native-sql-engine/pull/294)|[NSE-293] fix unsafemap with key = '0'|
46+
|[#233](https://github.com/oap-project/native-sql-engine/pull/233)|[NSE-207] fix issues found from aggregate unit tests|
47+
|[#246](https://github.com/oap-project/native-sql-engine/pull/246)|[NSE-245]Adding columnar RDD cache support|
48+
|[#289](https://github.com/oap-project/native-sql-engine/pull/289)|[NSE-206]Update installation guide and configuration guide.|
49+
|[#277](https://github.com/oap-project/native-sql-engine/pull/277)|[NSE-276] Add option to switch Hadoop version|
50+
|[#275](https://github.com/oap-project/native-sql-engine/pull/275)|[NSE-274] Comment to trigger tpc-h RAM test|
51+
|[#271](https://github.com/oap-project/native-sql-engine/pull/271)|[NSE-196] clean up configs in unit tests|
52+
|[#258](https://github.com/oap-project/native-sql-engine/pull/258)|[NSE-257] fix different versions of slf4j-log4j12|
53+
|[#259](https://github.com/oap-project/native-sql-engine/pull/259)|[NSE-248] fix arrow dependency order|
54+
|[#249](https://github.com/oap-project/native-sql-engine/pull/249)|[NSE-241] fix hashagg result length|
55+
|[#255](https://github.com/oap-project/native-sql-engine/pull/255)|[NSE-256] do not run ram report test on each PR|
56+
57+
58+
### SQL DS Cache
59+
60+
#### Features
61+
|||
62+
|:---|:---|
63+
|[#118](https://github.com/oap-project/sql-ds-cache/issues/118)|port to Spark 3.1.1|
64+
65+
#### Bugs Fixed
66+
|||
67+
|:---|:---|
68+
|[#121](https://github.com/oap-project/sql-ds-cache/issues/121)|OAP Index creation stuck issue|
69+
70+
#### PRs
71+
|||
72+
|:---|:---|
73+
|[#132](https://github.com/oap-project/sql-ds-cache/pull/132)|Fix SampleBasedStatisticsSuite UnitTest case|
74+
|[#122](https://github.com/oap-project/sql-ds-cache/pull/122)|[ sql-ds-cache-121] Fix Index stuck issues|
75+
|[#119](https://github.com/oap-project/sql-ds-cache/pull/119)|[SQL-DS-CACHE-118][POAE7-1130] port sql-ds-cache to Spark3.1.1|
76+
77+
78+
### OAP MLlib
79+
80+
#### Features
81+
|||
82+
|:---|:---|
83+
|[#26](https://github.com/oap-project/oap-mllib/issues/26)|[PIP] Support Spark 3.0.1 / 3.0.2 and upcoming 3.1.1|
84+
85+
#### PRs
86+
|||
87+
|:---|:---|
88+
|[#39](https://github.com/oap-project/oap-mllib/pull/39)|[ML-26] Build for different spark version by -Pprofile|
89+
90+
91+
### PMEM Spill
92+
93+
#### Features
94+
|||
95+
|:---|:---|
96+
|[#34](https://github.com/oap-project/pmem-spill/issues/34)|Support vanilla spark 3.1.1|
97+
98+
#### PRs
99+
|||
100+
|:---|:---|
101+
|[#41](https://github.com/oap-project/pmem-spill/pull/41)|[PMEM-SPILL-34][POAE7-1119]Port RDD cache to Spark 3.1.1 as separate module|
102+
103+
104+
### PMEM Common
105+
106+
#### Features
107+
|||
108+
|:---|:---|
109+
|[#10](https://github.com/oap-project/pmem-common/issues/10)|add -mclflushopt flag to enable clflushopt for gcc|
110+
|[#8](https://github.com/oap-project/pmem-common/issues/8)|use clflushopt instead of clflush |
111+
112+
#### PRs
113+
|||
114+
|:---|:---|
115+
|[#11](https://github.com/oap-project/pmem-common/pull/11)|[PMEM-COMMON-10][POAE7-1010]Add -mclflushopt flag to enable clflushop…|
116+
|[#9](https://github.com/oap-project/pmem-common/pull/9)|[PMEM-COMMON-8][POAE7-896]use clflush optimize version for clflush|
117+
118+
119+
### PMEM Shuffle
120+
121+
#### Features
122+
|||
123+
|:---|:---|
124+
|[#15](https://github.com/oap-project/pmem-shuffle/issues/15)|Doesn't work with Spark3.1.1|
125+
126+
#### PRs
127+
|||
128+
|:---|:---|
129+
|[#16](https://github.com/oap-project/pmem-shuffle/pull/16)|[pmem-shuffle-15] Make pmem-shuffle support Spark3.1.1|
130+
131+
132+
### Remote Shuffle
133+
134+
#### Features
135+
|||
136+
|:---|:---|
137+
|[#18](https://github.com/oap-project/remote-shuffle/issues/18)|upgrade to Spark-3.1.1|
138+
|[#11](https://github.com/oap-project/remote-shuffle/issues/11)|Support DAOS Object Async API|
139+
140+
#### PRs
141+
|||
142+
|:---|:---|
143+
|[#19](https://github.com/oap-project/remote-shuffle/pull/19)|[REMOTE-SHUFFLE-18] upgrade to Spark-3.1.1|
144+
|[#14](https://github.com/oap-project/remote-shuffle/pull/14)|[REMOTE-SHUFFLE-11] Support DAOS Object Async API|
145+
146+
3147

4148
## Release 1.1.0
5-
* [Native SQL Engine](#native-sql-engine)
6-
* [SQL DS Cache](#sql-ds-cache)
7-
* [OAP MLlib](#oap-mllib)
8-
* [PMEM Spill](#pmem-spill)
9-
* [PMEM Shuffle](#pmem-shuffle)
10-
* [Remote Shuffle](#remote-shuffle)
11149

12150
### Native SQL Engine
13151

@@ -264,7 +402,7 @@ Generated on 2021-04-29
264402
|[#6](https://github.com/oap-project/pmem-shuffle/pull/6)|[PMEM-SHUFFLE-7] enable fsdax mode in pmem-shuffle|
265403

266404

267-
### Remote-Shuffle
405+
### Remote Shuffle
268406

269407
#### Features
270408
|||

‎docs/OAP-Developer-Guide.md‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,13 @@
33
This document contains the instructions & scripts on installing necessary dependencies and building OAP modules.
44
You can get more detailed information from OAP each module below.
55

6-
* [SQL Index and Data Source Cache](https://github.com/oap-project/sql-ds-cache/blob/v1.1.0-spark-3.0.0/docs/Developer-Guide.md)
7-
* [PMem Common](https://github.com/oap-project/pmem-common/tree/v1.1.0-spark-3.0.0)
8-
* [PMem Spill](https://github.com/oap-project/pmem-spill/tree/v1.1.0-spark-3.0.0)
9-
* [PMem Shuffle](https://github.com/oap-project/pmem-shuffle/tree/v1.1.0-spark-3.0.0#5-install-dependencies-for-pmem-shuffle)
10-
* [Remote Shuffle](https://github.com/oap-project/remote-shuffle/tree/v1.1.0-spark-3.0.0)
11-
* [OAP MLlib](https://github.com/oap-project/oap-mllib/tree/v1.1.0-spark-3.0.0)
12-
* [Native SQL Engine](https://github.com/oap-project/native-sql-engine/tree/v1.1.0-spark-3.0.0)
6+
* [SQL Index and Data Source Cache](https://github.com/oap-project/sql-ds-cache/blob/v1.1.1-spark-3.1.1/docs/Developer-Guide.md)
7+
* [PMem Common](https://github.com/oap-project/pmem-common/tree/v1.1.1-spark-3.1.1)
8+
* [PMem Spill](https://github.com/oap-project/pmem-spill/tree/v1.1.1-spark-3.1.1)
9+
* [PMem Shuffle](https://github.com/oap-project/pmem-shuffle/tree/v1.1.1-spark-3.1.1#5-install-dependencies-for-pmem-shuffle)
10+
* [Remote Shuffle](https://github.com/oap-project/remote-shuffle/tree/v1.1.1-spark-3.1.1)
11+
* [OAP MLlib](https://github.com/oap-project/oap-mllib/tree/v1.1.1-spark-3.1.1)
12+
* [Native SQL Engine](https://github.com/oap-project/native-sql-engine/tree/v1.1.1-spark-3.1.1)
1313

1414
## Building OAP
1515

@@ -22,18 +22,18 @@ We provide scripts to help automatically install dependencies required, please c
2222
# cd oap-tools
2323
# sh dev/install-compile-time-dependencies.sh
2424
```
25-
*Note*: oap-tools tag version `v1.1.0-spark-3.0.0` corresponds to all OAP modules' tag version `v1.1.0-spark-3.0.0`.
25+
*Note*: oap-tools tag version `v1.1.1-spark-3.1.1` corresponds to all OAP modules' tag version `v1.1.1-spark-3.1.1`.
2626

2727
Then the dependencies below will be installed:
2828

29-
* [Cmake](https://help.directadmin.com/item.php?id=494)
29+
* [Cmake](https://cmake.org/install/)
3030
* [GCC > 7](https://gcc.gnu.org/wiki/InstallingGCC)
3131
* [Memkind](https://github.com/memkind/memkind/tree/v1.10.1)
3232
* [Vmemcache](https://github.com/pmem/vmemcache)
3333
* [HPNL](https://github.com/Intel-bigdata/HPNL)
3434
* [PMDK](https://github.com/pmem/pmdk)
3535
* [OneAPI](https://software.intel.com/content/www/us/en/develop/tools/oneapi.html)
36-
* [Arrow](https://github.com/oap-project/arrow/tree/arrow-3.0.0-oap-1.1)
36+
* [Arrow](https://github.com/oap-project/arrow/tree/arrow-4.0.0-oap-1.1.1)
3737
* [LLVM](https://llvm.org/)
3838

3939
Run the following command to learn more.

‎docs/OAP-Installation-Guide.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -29,17 +29,17 @@ Create a Conda environment and install OAP Conda package.
2929
```bash
3030
$ conda create -n oapenv -y python=3.7
3131
$ conda activate oapenv
32-
$ conda install -c conda-forge -c intel -y oap=1.1.0
32+
$ conda install -c conda-forge -c intel -y oap=1.1.1
3333
```
3434

3535
Once finished steps above, you have completed OAP dependencies installation and OAP building, and will find built OAP jars under `$HOME/miniconda2/envs/oapenv/oap_jars`
3636

3737
Dependencies below are required by OAP and all of them are included in OAP Conda package, they will be automatically installed in your cluster when you Conda install OAP. Ensure you have activated environment which you created in the previous steps.
3838

39-
- [Arrow](https://github.com/Intel-bigdata/arrow)
39+
- [Arrow](https://github.com/oap-project/arrow/tree/arrow-4.0.0-oap-1.1.1)
4040
- [Plasma](http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/)
41-
- [Memkind](https://anaconda.org/intel/memkind)
42-
- [Vmemcache](https://anaconda.org/intel/vmemcache)
41+
- [Memkind](https://github.com/memkind/memkind/tree/v1.10.1)
42+
- [Vmemcache](https://github.com/pmem/vmemcache.git)
4343
- [HPNL](https://anaconda.org/intel/hpnl)
4444
- [PMDK](https://github.com/pmem/pmdk)
4545
- [OneAPI](https://software.intel.com/content/www/us/en/develop/tools/oneapi.html)

‎mkdocs.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,5 +19,5 @@ theme: readthedocs
1919
plugins:
2020
- search
2121
- mkdocs-versioning:
22-
version: master
22+
version: 1.1.1
2323
exclude_from_nav: ["image", "js", "css", "fonts", "img"]

0 commit comments

Comments
 (0)