Merge pull request #1 from nodestream-proj/intial-version

Implement Connector

Author: zprobst

.github/CODEOWNERS

@@ -0,0 +1,3 @@
+#This is used to automatically add reviewers to PR's. Developers can still manually add reviewers in addition to the below users.
+
+* @zprobst @ccloes

.github/CONTRIBUTING.md

@@ -0,0 +1,3 @@
+# Contributing
+
+Review our Contributing Guide [in the Docs](https://nodestream-proj.github.io/nodestream/contributing).

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,24 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[BUG]"
+labels: bug
+assignees: zprobst
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: "[REQUEST]"
+labels: enhancement
+assignees: zprobst
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/workflows/ci.yaml

@@ -0,0 +1,45 @@
+name: Continuous Integration
+
+on: 
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  build:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: ["ubuntu-latest", "windows-latest", "macos-latest"]
+        python-version: ["3.10", "3.11"]
+    steps:
+      - uses: actions/checkout@v3
+      - name: Install poetry
+        run: pipx install poetry
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: 'poetry'
+      - name: Install Dependencies
+        run: |
+          pip install --upgrade pip
+          pip install --upgrade setuptools
+          poetry env use ${{ matrix.python-version }}
+          poetry install
+      - name: Run Tests
+        run: poetry run pytest --cov nodestream_plugin_neo4j --cov-report term --cov-report xml -m "not e2e"
+      - name: Run Lints
+        run: |
+          poetry run ruff nodestream_plugin_neo4j tests
+          poetry run black nodestream_plugin_neo4j tests --check
+          poetry run isort nodestream_plugin_neo4j tests --check-only
+      - name: Run E2E Tests
+        run: poetry run pytest -m "e2e"
+        if: "runner.os == 'Linux'"
+      - name: Upload coverage reports to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          flags: ${{ matrix.python-version }}-${{ matrix.os }}

.gitignore

@@ -0,0 +1,160 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/

.vscode/settings.json

@@ -0,0 +1,7 @@
+{
+    "python.testing.pytestArgs": [
+        "."
+    ],
+    "python.testing.unittestEnabled": false,
+    "python.testing.pytestEnabled": true
+}

Makefile

@@ -0,0 +1,56 @@
+SHELL := /bin/bash
+
+.PHONY: setup
+setup: venv lint test-unit
+
+.PHONY: clean-pyc
+clean-pyc:
+	find . -name '*.pyc' -exec rm -rf {} +
+	find . -name '*.pyo' -exec rm -rf {} +
+	find . -name '*~' -exec rm -rf {} +
+	find . -name '__pycache__' -exec rm -rf {} +
+
+.PHONY: clean-test
+clean-test:
+	rm -rf .pytest_cache
+	rm -rf .mypy_cache
+	rm -rf .coverage
+	rm -rf .reports
+	rm -rf htmlcov/
+	rm -rf .pytest_cache
+
+.PHONY: clean
+clean: clean-pyc clean-test
+
+venv: poetry.lock
+	poetry install
+
+.PHONY: format
+format: venv
+	poetry run black nodestream_plugin_neo4j tests
+	poetry run isort nodestream_plugin_neo4j tests
+
+.PHONY: lint
+lint: venv
+	poetry run black nodestream_plugin_neo4j tests --check
+	poetry run ruff nodestream_plugin_neo4j tests
+
+.PHONY: test-unit
+test-unit: venv
+	poetry run pytest -m "not e2e"
+
+.PHONY: test-e2e
+test-e2e: venv
+	poetry run pytest -m "e2e"
+
+.PHONY: docs-build
+docs-build: venv
+	poetry run mkdocs build
+
+.PHONY: docs-server
+docs-server: venv
+	poetry run mkdocs serve
+
+.PHONY: snapshot
+snapshot: venv
+	poetry run pytest --snapshot-update

nodestream_plugin_neo4j/__init__.py

@@ -0,0 +1,3 @@
+from .database_connector import Neo4jDatabaseConnector
+
+__all__ = ("Neo4jDatabaseConnector",)

nodestream_plugin_neo4j/database_connector.py

@@ -0,0 +1,54 @@
+from nodestream.databases import DatabaseConnector, TypeRetriever
+from nodestream.databases.query_executor import QueryExecutor
+from nodestream.schema.migrations import Migrator
+
+from .ingest_query_builder import Neo4jIngestQueryBuilder
+from .migrator import Neo4jMigrator
+from .neo4j_database import Neo4jDatabaseConnection
+from .query_executor import Neo4jQueryExecutor
+from .type_retriever import Neo4jTypeRetriever
+
+
+class Neo4jDatabaseConnector(DatabaseConnector, alias="neo4j"):
+    """A Connector for Neo4j Graph Databases.
+
+    This class is responsible for creating the various components needed for
+    nodestream to interact with a Neo4j database. It is also responsible
+    for providing the configuration options for the Neo4j database.
+    """
+
+    @classmethod
+    def from_file_data(
+        cls,
+        use_enterprise_features: bool = False,
+        use_apoc: bool = True,
+        **connection_args
+    ):
+        database_connection = Neo4jDatabaseConnection.from_configuration(
+            **connection_args
+        )
+        return cls(
+            database_connection=database_connection,
+            use_enterprise_features=use_enterprise_features,
+            use_apoc=use_apoc,
+        )
+
+    def __init__(
+        self,
+        database_connection: Neo4jDatabaseConnection,
+        use_apoc: bool,
+        use_enterprise_features: bool,
+    ) -> None:
+        self.use_enterprise_features = use_enterprise_features
+        self.use_apoc = use_apoc
+        self.database_connection = database_connection
+
+    def make_query_executor(self) -> QueryExecutor:
+        query_builder = Neo4jIngestQueryBuilder(self.use_apoc)
+        return Neo4jQueryExecutor(self.database_connection, query_builder)
+
+    def make_type_retriever(self) -> TypeRetriever:
+        return Neo4jTypeRetriever(self.database_connection)
+
+    def make_migrator(self) -> Migrator:
+        return Neo4jMigrator(self.database_connection, self.use_enterprise_features)