test: migrate e2e presubmit tests to bigframes-load-testing project (#160)

BEGIN_COMMIT_OVERRIDE
fix: migrate e2e tests to bigframes-load-testing project
END_COMMIT_OVERRIDE

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
- [ ] Make sure to open an issue as a [bug/issue](https://togithub.com/googleapis/python-bigquery-dataframes/issues/new/choose) before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)

Fixes internal issue 307809767 🦕

Author: shobsi

.kokoro/continuous/e2e.cfg

@@ -5,3 +5,13 @@ env_vars: {
     key: "NOX_SESSION"
     value: "unit_prerelease system_prerelease system_noextras e2e notebook samples"
 }
+
+env_vars: {
+    key: "GOOGLE_CLOUD_PROJECT"
+    value: "bigframes-load-testing"
+}
+
+env_vars: {
+    key: "BIGFRAMES_TEST_MODEL_VERTEX_ENDPOINT"
+    value: "https://us-central1-aiplatform.googleapis.com/v1/projects/272725758477/locations/us-central1/endpoints/590545496255234048"
+}

.kokoro/presubmit/e2e.cfg

@@ -5,3 +5,13 @@ env_vars: {
     key: "NOX_SESSION"
     value: "unit_prerelease system_prerelease system_noextras e2e notebook samples"
 }
+
+env_vars: {
+    key: "GOOGLE_CLOUD_PROJECT"
+    value: "bigframes-load-testing"
+}
+
+env_vars: {
+    key: "BIGFRAMES_TEST_MODEL_VERTEX_ENDPOINT"
+    value: "https://us-central1-aiplatform.googleapis.com/v1/projects/272725758477/locations/us-central1/endpoints/590545496255234048"
+}

CONTRIBUTING.rst

@@ -155,7 +155,44 @@ Running System Tests
   auth settings and change some configuration in your project to
   run all the tests.
 
-- System tests will be run against an actual project. You should use local credentials from gcloud when possible. See `Best practices for application authentication <https://cloud.google.com/docs/authentication/best-practices-applications#local_development_and_testing_with_the>`__. Some tests require a service account. For those tests see `Authenticating as a service account <https://cloud.google.com/docs/authentication/production>`__.
+- System tests will be run against an actual project. A project can be set in
+  the environment variable ``$GOOGLE_CLOUD_PROJECT``. If not, the project property
+  set in the `Google Cloud CLI <https://cloud.google.com/sdk/gcloud/reference/config/get>`__
+  will be effective, which can be peeked into via ``gcloud config get project``,
+  or set via ``gcloud config set project <project-name>``. The following roles
+  carry the permissions to run the system tests in the project:
+
+  - `BigQuery User <https://cloud.google.com/bigquery/docs/access-control#bigquery.user>`__
+    to be able to create test datasets and run BigQuery jobs in the project.
+
+  - `BigQuery Connection Admin <https://cloud.google.com/bigquery/docs/access-control#bigquery.connectionAdmin>`__
+    to be able to use BigQuery connections in the project.
+
+  - `BigQuery Data Editor <https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor>`__
+    to be able to create BigQuery remote functions in the project.
+
+  - `Browser <https://cloud.google.com/resource-manager/docs/access-control-proj#browser>`__
+    to be able to get current IAM policy for the service accounts of the BigQuery connections in the project.
+
+  - `Cloud Functions Developer <https://cloud.google.com/functions/docs/reference/iam/roles#cloudfunctions.developer>`__
+    to be able to create cloud functions to support BigQuery DataFrames remote functions.
+
+  - `Service Account User <https://cloud.google.com/iam/docs/service-account-permissions#user-role>`__
+    to be able to use the project's service accounts.
+
+  - `Vertex AI User <https://cloud.google.com/vertex-ai/docs/general/access-control#aiplatform.user>`__
+    to be able to use the BigQuery DataFrames' ML integration with Vertex AI.
+
+- You can run the script ``scripts/setup-project-for-testing.sh <project-id> [<principal>]``
+  to set up a project for running system tests and optionally set up necessary
+  IAM roles for a principal (user/group/service-account). You need to have the following
+  IAM permission to be able to run the set up script successfully:
+
+  - ``serviceusage.services.enable``
+  - ``bigquery.connections.create``
+  - ``resourcemanager.projects.setIamPolicy``
+
+- You should use local credentials from gcloud when possible. See `Best practices for application authentication <https://cloud.google.com/docs/authentication/best-practices-applications#local_development_and_testing_with_the>`__. Some tests require a service account. For those tests see `Authenticating as a service account <https://cloud.google.com/docs/authentication/production>`__.
 
 *************
 Test Coverage

bigframes/remote_function.py

@@ -411,13 +411,23 @@ def create_cloud_function(self, def_, cf_name, package_requirements=None):
             create_function_request.function = function
 
             # Create the cloud function and wait for it to be ready to use
-            operation = self._cloud_functions_client.create_function(
-                request=create_function_request
-            )
-            operation.result()
-
-            # Cleanup
-            os.remove(archive_path)
+            try:
+                operation = self._cloud_functions_client.create_function(
+                    request=create_function_request
+                )
+                operation.result()
+
+                # Cleanup
+                os.remove(archive_path)
+            except google.api_core.exceptions.AlreadyExists:
+                # If a cloud function with the same name already exists, let's
+                # update it
+                update_function_request = functions_v2.UpdateFunctionRequest()
+                update_function_request.function = function
+                operation = self._cloud_functions_client.update_function(
+                    request=update_function_request
+                )
+                operation.result()
 
         # Fetch the endpoint of the just created function
         endpoint = self.get_cloud_function_endpoint(cf_name)

notebooks/generative_ai/large_language_models.ipynb

@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -22,12 +22,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "session = bigframes.pandas.get_global_session()\n",
-    "connection = \"bigframes-dev.us.bigframes-ml\""
+    "connection = f\"{session.bqclient.project}.us.bigframes-default-connection\""
    ]
   },
   {