Skip to content

Commit 3b718e4

Browse files
Merge pull request #52 from vertica/s3x-master
S3x master
2 parents 4fc5a42 + f6b9abf commit 3b718e4

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

66 files changed

+4019
-231
lines changed

.gitconfig

Lines changed: 0 additions & 3 deletions
This file was deleted.

.github/workflows/python-publish.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,3 +29,4 @@ jobs:
2929
run: |
3030
python setup.py sdist bdist_wheel
3131
twine upload dist/*
32+

.github/workflows/vertica-test.yml

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,20 @@ jobs:
4242
run: pip install -r requirements.txt
4343
- name: Install
4444
run: python setup.py install
45-
- name: Test
46-
run: python -m pytest tests/functional
45+
- name: Test Basic
46+
run: python -m pytest tests/functional/adapter/test_basic.py
47+
- name: Test Concurrency
48+
run: python -m pytest tests/functional/adapter/concurrency/
49+
- name: Test ephemeral
50+
run: python -m pytest tests/functional/adapter/ephemeral/
51+
- name: Test Doc Generate
52+
run: python -m pytest tests/functional/adapter/test_doc_gen.py
53+
- name: Test Data Type Boolean
54+
run: python -m pytest tests/functional/adapter/test_data_types.py
55+
- name: Test Concat
56+
run: python -m pytest tests/functional/adapter/utils/test_concat.py
57+
- name: Test Data Type Int
58+
run: python -m pytest tests/functional/adapter/utils/data_type/
4759

4860

4961

.gitignore

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
1-
.vscode/
2-
*.pyc
3-
*.egg-info/
4-
dist/
51
build/
6-
requirements.txt
7-
venv/
2+
logs
3+
*.egg-info
4+
*__pycache__
5+
*.pyc

CHANGLOG.md

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
## Changelog
2+
- This file provides a full account of all changes to dbt-vertica.
3+
- "Breaking changes" listed under a version may require action from end users.
4+
### 1.3.0
5+
#### Features:
6+
- Added support for [`dbt-core version 1.3.0`](https://github.com/dbt-labs/dbt-core/discussions/6011) and migrated testing framework to new testing framework according to DBT guidelines.
7+
- Support for incremental model strategy ‘Append’.
8+
- Support for incremental model strategy ‘insert_overwrite’.
9+
- Support for multiple optimization parameters for table materialization:
10+
- order_by
11+
- segmented_by_string
12+
- segmented_by_all_nodes
13+
- no_segmentation
14+
- ksafe
15+
- partition_by_string
16+
- partition_by_group_by_string
17+
- partition_by_active_count
18+
- Support for enabling privileges inheritance for tables/views using INCLUDE SCHEMA PRIVILEGES by default in model materialization. If not required, can be disabled using EXCLUDE in the Vertica Server.
19+
- Defined profile_template which helps user to configure profile while creating the project.
20+
- Support for Python 3.10.
21+
#### Fixes:
22+
- Incremental materialization refactoring and cleanup.
23+
- Updates to correctly handle errors for multi-statement queries.
24+
#### Breaking Changes
25+
##### Change description:
26+
- Refactored `merge_columns` config parameter to `unique_key`.
27+
- Support for the `merge_update_columns` to only merge the columns specified.
28+
##### Impact:
29+
- For the incremental model strategy ‘delete+insert’ and ‘merge’, `unique_key` is now a required parameter and it fails if not provided.
30+
- Existing applications using config parameter `merge_columns` will give an error because `merge_columns` as been removed.
31+
##### Workaround/Solution:
32+
- When using the incremental model strategy ‘delete+insert’ and ‘merge’ pass the required parameter `unique_key` instead of `merge_columns` in config and `merge_update_columns` is used to only merge the columns specified.
33+
### 1.0.3
34+
- Refactored the adapter to model after dbt's global_project macros
35+
- Unimplemented functions should throw an exception that it's not implemented. If you stumble across this, please open an Issue or PR so we can investigate.
36+
### 1.0.2
37+
- Added support for snapshot timestamp with passing tests
38+
- Added support for snapshot check cols with passing tests
39+
### 1.0.1
40+
- Fixed the Incremental method implementation (was buggy/incomplete)
41+
- Removed the `unique_id` as it wasn't implemented
42+
- Fixed when no fields were added - full table merge
43+
- Added testing for Incremental materialization
44+
- Testing for dbt Incremental full table
45+
- Testing for dbt Incremental specified merged columns
46+
- Added more logging to the connector to help understand why tests were failing
47+
- Using the official [Vertica CE 11.0.x docker image](https://hub.docker.com/r/vertica/vertica-ce) now for tests
48+
### 1.0.0
49+
- Add support for DBT version 1.0.0
50+
### 0.21.1
51+
- Add testing, fix schema drop.
52+
### 0.21.0
53+
- Add `unique_field` property on connection, supporting 0.21.x.
54+
### 0.20.2
55+
- Added SSL options.
56+
### 0.20.1
57+
- Added the required changes from dbt 0.19.0. [Details found here](https://docs.getdbt.com/docs/guides/migration-guide/upgrading-to-0-19-0#for-dbt-plugin-maintainers).
58+
- Added support for the MERGE command for incremental loading isntead of DELETE+INSERT

CONTRIBUTING.md

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
Firstly, we appreciate you thinking about contributing to dbt-vertica so that it can continue to evolve.
2+
This document will guide you through the contribution process. There are several ways you can help:
3+
4+
- [Bug Reports](#bug-reports)
5+
- [Feature Requests](#feature-requests)
6+
- [Code Contributions](#code-contributions)
7+
8+
# Bug Reports
9+
10+
If you find a bug, submit an [issue](https://github.com/vertica/dbt-vertica/issues) with a complete and reproducible bug report. If the issue can't be reproduced, it will be closed. If you opened an issue, but figured out the answer later on your own, comment on the issue to let people know, then close the issue.
11+
12+
For issues (e.g. security related issues) that are **not suitable** to be reported publicly on the GitHub issue system, report your issues to [Vertica open source team](mailto:vertica-opensrc@microfocus.com) directly or file a case with Vertica support if you have a support account.
13+
14+
# Feature Requests
15+
16+
Feel free to share your ideas for how to improve *dbt-vertica*. We’re always open to suggestions.
17+
You can open an [issue](https://github.com/vertica/dbt-vertica/issues)
18+
with details describing what feature(s) you'd like added or changed.
19+
20+
If you would like to implement the feature yourself, open an issue to ask before working on it. Once approved, please refer to the [Code Contributions](#code-contributions) section.
21+
22+
# Code Contributions
23+
24+
## Step 1: Fork
25+
Fork the project [on Github](https://github.com/vertica/dbt-vertica) and check out your copy locally.
26+
```shell
27+
git clone git@github.com:YOURUSERNAME/dbt-vertica.git
28+
cd dbt-vertica
29+
```
30+
Your GitHub repository **YOURUSERNAME/dbt-vertica**r will be called "origin" in Git.
31+
You should also set up **vertica/dbt-vertica** as an "upstream" remote.
32+
33+
```shell
34+
git remote add upstream git@github.com:vertica/dbt-vertica.git
35+
git fetch upstream
36+
```
37+
### Configure Git for the first time
38+
39+
Make sure git knows your [name](https://help.github.com/articles/setting-your-username-in-git/ "Set commit username in Git") and [email address](https://help.github.com/articles/setting-your-commit-email-address-in-git/ "Set commit email address in Git"):
40+
41+
```shell
42+
git config --global user.name "John Smith"
43+
git config --global user.email "email@example.com"
44+
```
45+
46+
## Step 2: Branch
47+
48+
Create a new branch for the work with a descriptive name:
49+
```shell
50+
git checkout -b my-fix-branch
51+
```
52+
53+
## Step 3: Install dependencies
54+
55+
Install the dbt-vertica dependencies for development:
56+
```shell
57+
58+
pip install dbt-vertica
59+
60+
```
61+
62+
## Step 4: Get the test suite running
63+
64+
*dbt-vertica* comes with a test suite of its own, in the Tests directory of the code base. It’s our policy to make sure all tests always pass.
65+
We appreciate any and all contributions to the test suite! These tests use a Pytest framework: testing. You might want to check out the testing documentation for more details.
66+
67+
Syntax tests do simple syntax testing of individual metrics and configs, which do not require database connection. Examples of running tests:
68+
69+
Run all test :
70+
```shell
71+
$ cd /tests
72+
```
73+
```shell
74+
$ pytest functional/adapter/
75+
```
76+
77+
The Github Actions CI workflow committed as part of the project will automatically run test suite through different pytest versions. These CI tests must pass before any PR will be considered. This CI workflow can be run on your forked repository after you are enabling Github Actions on your fork.
78+
79+
## Step 5: Implement your fix or feature
80+
81+
At this point, you're ready to make your changes! Feel free to ask for help; everyone is a beginner at first.
82+
## License Headers
83+
Every file in this project must use the following Apache 2.0 header (with the appropriate year or years in the "[yyyy]" box; if a copyright statement from another party is already present in the code, you may add the statement on top of the existing copyright statement):
84+
85+
```
86+
Copyright (c) [yyyy] Micro Focus or one of its affiliates.
87+
Licensed under the Apache License, Version 2.0 (the "License");
88+
you may not use this file except in compliance with the License.
89+
You may obtain a copy of the License at
90+
http://www.apache.org/licenses/LICENSE-2.0
91+
Unless required by applicable law or agreed to in writing, software
92+
distributed under the License is distributed on an "AS IS" BASIS,
93+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
94+
See the License for the specific language governing permissions and
95+
limitations under the License.
96+
```
97+
98+
### Commits
99+
Make some changes on your branch, then stage and commit as often as necessary:
100+
```shell
101+
git add.
102+
git commit -m 'Added two more tests for #166'
103+
```
104+
When writing the commit message, try to describe precisely what the commit does. The commit message should be in lines of 72 chars maximum. Include the issue number #N if the commit is related to an issue.
105+
106+
### Tests
107+
Add appropriate tests for the bug’s or feature's behavior, run the test suite again and ensure that all tests pass. Here is the guideline for writing test:
108+
- Tests should be easy for any contributor to run. Contributors may not get complete access to their Vertica database, for example, they may only have a non-admin user with write privileges to a single schema, and the database may not be the latest version. We encourage tests to use only what they need and nothing more.
109+
- If there are requirements to the database for running a test, the test should adapt to different situations and never report a failure. For example, if a test depends on a multi-node database, it should check the number of DB nodes first, and skip itself when it connects to a single-node database.
110+
111+
## Step 6: Push and Rebase
112+
113+
You can publish your work on GitHub just by doing:
114+
```shell
115+
$ git push origin my-fix-branch
116+
```
117+
When you go to your GitHub page, you will notice commits made on your local branch is pushed to the remote repository.
118+
When upstream (vertica/dbt-vertica) has changed, you should **rebase** your work. The rebase command creates a linear history by moving your local commits onto the tip of the upstream commits.
119+
120+
You can rebase your branch locally and force-push to your GitHub repository by doing:
121+
122+
```shell
123+
git checkout my-fix-branch
124+
git fetch upstream
125+
git rebase upstream/master
126+
git push -f origin my-fix-branch
127+
```
128+
129+
## Step 7: Make a Pull Request
130+
131+
When you think your work is ready to be pulled into dbt-vertica, you should create a pull request (PR) at GitHub.
132+
A good pull request means:
133+
134+
- commits with one logical change in each
135+
- well-formed messages for each commit
136+
- documentation and tests, if needed
137+
- Go to https://github.com/YOURUSERNAME/dbt-vertica and [make a Pull Request](https://help.github.com/articles/creating-a-pull-request/) to vertica:master.
138+
139+
**Note: When you initate a contribution some background tests run to verify you proposed changes pass. These tests can take 5 to 10 minutes to execute, so be patient. It is out there doing something when you see the pause in activity.**
140+
141+
### Sign the CLA
142+
143+
Before we can accept a pull request, we first ask people to sign a Contributor License Agreement (or CLA). We ask this so that we know that contributors have the right to donate the code. You should notice a comment from **CLAassistant** on your pull request page, follow this comment to sign the CLA electronically.
144+
145+
### Review
146+
147+
Pull requests are usually reviewed within a few days. If there are comments to address, apply your changes in new commits, rebase your branch and force-push to the same branch, re-run the test suite to ensure tests are still passing. We care about quality, Vertica has internal test suites to run as well, so your pull request won't be merged until all internal tests pass. To produce a clean commit history, our maintainers would do squash merging once your PR is approved, which means combining all commits of your PR into a single commit in the master branch.
148+
149+
You are done! Thank you for your code contribution!
150+
151+
After your pull request is merged, you can safely delete your branch and pull the changes from the upstream repository.

0 commit comments

Comments
 (0)