0.3.0

wangxiaoying · wangxiaoying · commit ea2e21a38ef4 · 2022-05-10T20:48:25.000-07:00
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/README.md b/README.md
@@ -32,14 +32,28 @@ The function will partition the query by **evenly** splitting the specified colu
 ConnectorX will assign one thread for each partition to load and write data in parallel.
 Currently, we support partitioning on **numerical** columns (**cannot contain NULL**) for **SPJA** queries. 
 
-Check out more detailed usage and examples [here](#detailed-usage-and-examples). A general introduction of the project can be found in this [blog post](https://towardsdatascience.com/connectorx-the-fastest-way-to-load-data-from-databases-a65d4d4062d5).
+
+We are now providing federated query support (experimental, PostgreSQL only and do not support partition for now), you can write a single query to join tables from two postgres databases. JRE (Java Runtime Environment) is required.
+
+```python
+import connectorx as cx
+
+db1 = "postgresql://username1:password1@server1:port1/database1"
+db2 = "postgresql://username2:password2@server2:port2/database2"
+
+cx.read_sql({"db1": db1, "db2": db2}, "SELECT * FROM db1.nation n, db2.region r where n.n_regionkey = r.r_regionkey")
+```
+
+Check out more detailed usage and examples [here](https://sfu-db.github.io/connector-x/api.html). A general introduction of the project can be found in this [blog post](https://towardsdatascience.com/connectorx-the-fastest-way-to-load-data-from-databases-a65d4d4062d5).
 
 # Installation
 
 ```bash
 pip install connectorx
 ```
 
+Check out [here](https://sfu-db.github.io/connector-x/install.html#build-from-source-code) to see how to build python wheel from source.
+
 # Performance
 
 We compared different solutions in Python that provides the `read_sql` function, by loading a 10x TPC-H lineitem table (8.6GB) from Postgres into a DataFrame, with 4 cores parallelism.
@@ -76,18 +90,10 @@ Finally, ConnectorX will use the schema info as well as the count info to alloca
 Once the downloading begins, there will be one thread for each partition so that the data are downloaded in parallel at the partition level. The thread will issue the query of the corresponding
 partition to the database and then write the returned data to the destination row-wise or column-wise (depends on the database) in a streaming fashion. 
 
-#### How to specify the partition number?
-
-`partition_num` will determine how many queries we are going to split from the original one and issue to the database. Underlying, we use [rayon](https://github.com/rayon-rs/rayon) as our parallel executor, which adopts a pool of threads to handle each partitioned query. The number of threads in the pool equals to the number of logical cores on the machine. It is recommended to set the `partition_num` to the number of available logical cores.
-
-#### How to choose the partition column?
-
-`partition_on` specifies on which column we will do the partition as above procedure. In order to achieve the best performance, it is ideal that each partitioned query will return the same number of rows. And since we partition the column evenly, it is recommended that the numerical `partition_on` column is evenly distributed. Whether a column has index or not might also affect the performance depends on the source database. You can give it a try if you have multiple candidates. Also, you can manually partition the query if our partition method cannot match your need. ConnectorX will still return a whole dataframe with all the results of the list of queries you input.
-
 
 # Supported Sources & Destinations
 
-Supported protocols, data types and type mappings can be found [here](Types.md).
+Example connection string, supported protocols and data types for each data source can be found [here](https://sfu-db.github.io/connector-x/databases.html).
 For more planned data sources, please check out our [discussion](https://github.com/sfu-db/connector-x/discussions/61).
 
 ## Sources
@@ -100,7 +106,7 @@ For more planned data sources, please check out our [discussion](https://github.
 - [x] SQL Server
 - [x] Azure SQL Database (through mssql protocol)
 - [x] Oracle
-- [x] Big Query - Experimental: need docs and benchmark (also more tests)
+- [x] Big Query
 - [ ] ODBC (WIP)
 - [ ] ...
 
@@ -110,95 +116,11 @@ For more planned data sources, please check out our [discussion](https://github.
 - [x] Modin (through Pandas)
 - [x] Dask (through Pandas)
 - [x] Polars (through PyArrow)
-  
-# Detailed Usage and Examples
-
-Rust docs: [stable](https://docs.rs/connectorx) [nightly](https://sfu-db.github.io/connector-x/connectorx/)
-
-## API
-
-```python
-connectorx.read_sql(conn: str, query: Union[List[str], str], *, return_type: str = "pandas", protocol: str = "binary", partition_on: Optional[str] = None, partition_range: Optional[Tuple[int, int]] = None, partition_num: Optional[int] = None)
-```
 
-Run the SQL query, download the data from database into a Pandas dataframe.
+# Documentation
 
-## Parameters
-- `conn: str`: Connection string URI.
-  - General supported URI scheme: `(postgres|postgressql|mysql|mssql)://username:password@addr:port/dbname`.
-  - For now sqlite only support absolute path, example: `sqlite:///home/user/path/test.db`.
-  - Google BigQuery requires absolute path of the authentication JSON file, example: `bigquery:///home/user/path/auth.json`
-  - Please check out [here](Types.md) for more connection uri parameters supported for each database (e.g. `trusted_connection` and `encrypt` for Mssql, `sslmode` for Postgres)
-- `query: Union[str, List[str]]`: SQL query or list of SQL queries for fetching data.
-- `return_type: str = "pandas"`: The return type of this function. It can be `arrow`, `pandas`, `modin`, `dask` or `polars`.
-- `protocol: str = "binary"`: The protocol used to fetch data from source, default is `binary`. Check out [here](Types.md) to see more details.
-- `partition_on: Optional[str]`: The column to partition the result.
-- `partition_range: Optional[Tuple[int, int]]`: The value range of the partition column.
-- `partition_num: Optional[int]`: The number of partitions to generate.
-- `index_col: Optional[str]`: The index column to set for the result dataframe. Only applicable when `return_type` is `pandas`, `modin` or `dask`. 
-
-## Examples
-- Read a DataFrame from a SQL using a single thread
-
-  ```python
-  import connectorx as cx
-
-  postgres_url = "postgresql://username:password@server:port/database"
-  query = "SELECT * FROM lineitem"
-
-  cx.read_sql(postgres_url, query)
-  ```
-
-- Read a DataFrame parallelly using 10 threads by automatically partitioning the provided SQL on the partition column (`partition_range` will be automatically  queried if not given)
-
-  ```python
-  import connectorx as cx
-
-  postgres_url = "postgresql://username:password@server:port/database"
-  query = "SELECT * FROM lineitem"
-
-  cx.read_sql(postgres_url, query, partition_on="l_orderkey", partition_num=10)
-  ```
-
-- Read a DataFrame parallelly using 2 threads by manually providing two partition SQLs (the schemas of all the query results should be same)
-
-  ```python
-  import connectorx as cx
-
-  postgres_url = "postgresql://username:password@server:port/database"
-  queries = ["SELECT * FROM lineitem WHERE l_orderkey <= 30000000", "SELECT * FROM lineitem WHERE l_orderkey > 30000000"]
-
-  cx.read_sql(postgres_url, queries)
-
-  ```
-  
-- Read a DataFrame parallelly using 4 threads from a more complex query
-
-  ```python
-  import connectorx as cx
-
-  postgres_url = "postgresql://username:password@server:port/database"
-  query = f"""
-  SELECT l_orderkey,
-         SUM(l_extendedprice * ( 1 - l_discount )) AS revenue,
-         o_orderdate,
-         o_shippriority
-  FROM   customer,
-         orders,
-         lineitem
-  WHERE  c_mktsegment = 'BUILDING'
-         AND c_custkey = o_custkey
-         AND l_orderkey = o_orderkey
-         AND o_orderdate < DATE '1995-03-15'
-         AND l_shipdate > DATE '1995-03-15'
-  GROUP  BY l_orderkey,
-            o_orderdate,
-            o_shippriority 
-  """
-
-  cx.read_sql(postgres_url, query, partition_on="l_orderkey", partition_num=4)
-
-  ```
+Doc: https://sfu-db.github.io/connector-x/intro.html
+Rust docs: [stable](https://docs.rs/connectorx) [nightly](https://sfu-db.github.io/connector-x/connectorx/)
 
 # Next Plan
 
diff --git a/connectorx-python/Cargo.lock b/connectorx-python/Cargo.lock
diff --git a/connectorx-python/Cargo.toml b/connectorx-python/Cargo.toml
@@ -2,7 +2,7 @@
 authors = ["Weiyuan Wu <youngw@sfu.ca>"]
 edition = "2018"
 name = "connectorx-python"
-version = "0.2.6-alpha.6"
+version = "0.3.0"
 
 [workspace]
 # prevents package from thinking it's in the workspace
diff --git a/connectorx-python/pyproject.toml b/connectorx-python/pyproject.toml
@@ -18,7 +18,7 @@ license = "MIT"
 maintainers = ["Weiyuan Wu <youngw@sfu.ca>"] 
 name = "connectorx" 
 readme = "README.md" # Markdown files are supported
-version = "0.2.6-alpha.6" 
+version = "0.3.0" 
 
 [project]
 name = "connectorx" # Target file name of maturin build
diff --git a/connectorx/Cargo.toml b/connectorx/Cargo.toml
@@ -7,7 +7,7 @@ license = "MIT"
 name = "connectorx"
 readme = "../README.md"
 repository = "https://github.com/sfu-db/connector-x"
-version = "0.2.6-alpha.6"
+version = "0.3.0"
 
 [dependencies]
 anyhow = "1"
diff --git a/docs/_toc.yml b/docs/_toc.yml
@@ -6,7 +6,7 @@ root: intro
 
 chapters:
   - file: install
-  # - title: Databases
+  - file: api
   - file: databases
     sections:
     - file: databases/bigquery
diff --git a/docs/api.md b/docs/api.md
@@ -0,0 +1,98 @@
+# Basic usage
+ConnectorX enables you to run the SQL query, load data from databases into a Pandas Dataframe in the fastest and most memory efficient way.
+
+## API
+```python
+connectorx.read_sql(conn: Union[str, Dict[str, str]], query: Union[List[str], str], *, return_type: str = "pandas", protocol: str = "binary", partition_on: Optional[str] = None, partition_range: Optional[Tuple[int, int]] = None, partition_num: Optional[int] = None)
+```
+
+## Parameters
+- `conn: Union[str, Dict[str, str]]`: Connection string URI for querying single database or dict of database names (key) and connection string URIs (value) for querying multiple databases.
+  - Please check out [here](https://sfu-db.github.io/connector-x/databases.html) for connection string examples of each database
+- `query: Union[str, List[str]]`: SQL query or list of partitioned SQL queries for fetching data.
+- `return_type: str = "pandas"`: The return type of this function. It can be `arrow` (`arrow2`), `pandas`, `modin`, `dask` or `polars`.
+- `protocol: str = "binary"`: The protocol used to fetch data from source, default is `binary`. Check out [here](./databases.md) to see more details.
+- `partition_on: Optional[str]`: The column to partition the result.
+- `partition_range: Optional[Tuple[int, int]]`: The value range of the partition column.
+- `partition_num: Optioinal[int]`: The number of partitions to generate.
+- `index_col: Optioinal[str]`: The index column to set for the result dataframe. Only applicable when `return_type` is `pandas`, `modin` or `dask`. 
+
+
+## Examples
+- Read a DataFrame from a SQL using a single thread
+
+  ```python
+  import connectorx as cx
+
+  postgres_url = "postgresql://username:password@server:port/database"
+  query = "SELECT * FROM lineitem"
+
+  cx.read_sql(postgres_url, query)
+  ```
+
+- Read a DataFrame parallelly using 10 threads by automatically partitioning the provided SQL on the partition column (`partition_range` will be automatically  queried if not given)
+
+  ```python
+  import connectorx as cx
+
+  postgres_url = "postgresql://username:password@server:port/database"
+  query = "SELECT * FROM lineitem"
+
+  cx.read_sql(postgres_url, query, partition_on="l_orderkey", partition_num=10)
+  ```
+
+- Read a DataFrame parallelly using 2 threads by manually providing two partition SQLs (the schemas of all the query results should be same)
+
+  ```python
+  import connectorx as cx
+
+  postgres_url = "postgresql://username:password@server:port/database"
+  queries = ["SELECT * FROM lineitem WHERE l_orderkey <= 30000000", "SELECT * FROM lineitem WHERE l_orderkey > 30000000"]
+
+  cx.read_sql(postgres_url, queries)
+
+  ```
+  
+- Read a DataFrame parallelly using 4 threads from a more complex query
+
+  ```python
+  import connectorx as cx
+
+  postgres_url = "postgresql://username:password@server:port/database"
+  query = f"""
+  SELECT l_orderkey,
+         SUM(l_extendedprice * ( 1 - l_discount )) AS revenue,
+         o_orderdate,
+         o_shippriority
+  FROM   customer,
+         orders,
+         lineitem
+  WHERE  c_mktsegment = 'BUILDING'
+         AND c_custkey = o_custkey
+         AND l_orderkey = o_orderkey
+         AND o_orderdate < DATE '1995-03-15'
+         AND l_shipdate > DATE '1995-03-15'
+  GROUP  BY l_orderkey,
+            o_orderdate,
+            o_shippriority 
+  """
+
+  cx.read_sql(postgres_url, query, partition_on="l_orderkey", partition_num=4)
+
+  ```
+
+- Read a DataFrame from a SQL joined from multiple databases (experimental, only support PostgreSQL for now)
+
+  ```python
+  import connectorx as cx
+
+  import connectorx as cx
+
+  db1 = "postgresql://username1:password1@server1:port1/database1"
+  db2 = "postgresql://username2:password2@server2:port2/database2"
+  query = "SELECT * FROM db1.nation n, db2.region r where n.n_regionkey = r.r_regionkey"
+
+  cx.read_sql({"db1": db1, "db2": db2}, query)
+
+  ```
+
diff --git a/docs/install.md b/docs/install.md
