Merge pull request #4710 from linuxfoundation/unicron-research-which-v1-and-v2-apis-are-still-used

Utils to lookup for V1/V2 Py APIs still used on dev & prod

Author: lukaszgryglicki

.gitignore

@@ -245,3 +245,4 @@ cla-backend/run-python-test-example-*.py
 # LG
 out
 *.secret
+*log*.json

docs/Python_APIs.md

@@ -0,0 +1,78 @@
+# V1/V2 python APIs still used
+
+1. EasyCLA backend [here](https://github.com/linuxfoundation/easycla/blob/main/.github/workflows/deploy-prod.yml#L127).
+- `/v2/health`. Health check while `dev`/`prod` deployment.
+
+
+2. EasyCLA contributor console [here](https://github.com/communitybridge/easycla-contributor-console/blob/main/src/app/core/services/cla-contributor.service.ts), found via `` find.sh . '*' /vN ``.
+- `/v1/api/health` [here](https://github.com/communitybridge/easycla-contributor-console/blob/main/test/functional/cypress/integration/api-tests/health-check.spec.ts#L4).
+- `/v1/user/gerrit`.
+- `/v2/user/<userId>`.
+- `/v2/project/<projectId>`.
+- `/v2/user/<userId>/active-signature`.
+- `/v2/check-prepare-employee-signature`.
+- `/v2/request-employee-signature`.
+- `/v2/user/<userId>/project/<projectId>/last-signature`.
+- `/v2/project/<projectId>`.
+
+
+3. EasyCLA corporate console [here](https://github.com/LF-Engineering/lfx-corp-cla-console/blob/main/backend/src/data/cla-api.ts):
+- No V1 or V2 APIs used.
+
+
+4. PCC [here](https://github.com/linuxfoundation/lfx-pcc/blob/main/apps/v1-backend/src/modules/cla-services/model/index.ts):
+- No V1 or V2 APIs used.
+
+
+5. GitHub/Gitlab/Gerrit ({provider}) app/bot [here]() (there is no list of which particular APIs are used by GitHub/GitLab/Gerrit):
+- `/v2/github/activity`.
+- `/v2/repository-provider/{provider}/sign/{installation_id}/{github_repository_id}/{change_request_id}`.
+- `/v2/github/installation`.
+- `/v1/user/gerrit`.
+- `/v2/signed/individual/{installation_id}/{github_repository_id}/{change_request_id}` ?
+- `/v2/repository-provider/{provider}/activity` ?
+- `/v2/repository-provider/{provider}/oauth2_redirect` ?
+- `/v2/signed/gitlab/individual/{user_id}/{organization_id}/{gitlab_repository_id}/{merge_request_id}` ?
+- `/v2/signed/gerrit/individual/{user_id}` ?
+- `cla-backend/cla/routes.py`: `GitHub Routes`, `Gerrit Routes`, `Gerrit instance routes`.
+
+
+6. Manually check which APIs were actually called on `dev` and `prod` via:
+
+- `prod` analysis: `` DEBUG=1 NO_ECHO=1 STAGE=prod REGION=us-east-1 DTFROM='10 days ago' DTTO='1 second ago' OUT=api-logs-prod.json ./utils/search_aws_logs.sh 'LG:api-request-path' && jq -r '.[].message' api-logs-prod.json | grep -o 'LG:api-request-path:[^[:space:]]*' | sed 's/^LG:api-request-path://' | sed -E 's/[0-9a-fA-F-]{36}/<uuid>/g' | sed -E 's/\b[0-9]{2,}\b/<id>/g' | sort | uniq -c | sort -nr ``:
+```
+ 377727 /v2/github/activity
+   1783 /v2/repository-provider/github/sign/<id>/<id>/<id>
+    347 /v2/github/installation
+    212 /v2/project/<uuid>
+    211 /v2/user/<uuid>
+    187 /v2/user/<uuid>/active-signature
+    157 /v2/check-prepare-employee-signature
+    146 /v2/return-url/<uuid>
+     61 /v2/request-employee-signature
+     16 /v1/file/icon/seo/<uuid>.png
+      8 /v2/gerrit/<uuid>/corporate/agreementUrl.html
+      7 /v1/user/gerrit
+      2 /v2/health
+      1 /v2/user-from-token
+      1 /v2/repository-provider/github/icon.svg
+```
+
+- `dev` analysis (but this can contain API calls made by developer and not actually used): `` DEBUG=1 STAGE=dev REGION=us-east-1 DTFROM='10 days ago' DTTO='1 second ago' OUT=api-logs-dev.json ./utils/search_aws_logs.sh 'LG:api-request-path' && jq -r '.[].message' api-logs-prod.json | grep -o 'LG:api-request-path:[^[:space:]]*' | sed 's/^LG:api-request-path://' | sed -E 's/[0-9a-fA-F-]{36}/<uuid>/g' | sed -E ':a;s#/([0-9]{1,})(/|$)#/<id>\2#g;ta' | sort | uniq -c | sort -nr ``:
+```
+    113 /v2/project/<uuid>
+     53 /v2/user/<uuid>
+     39 /v2/github/activity
+     14 /v2/user/<uuid>/active-signature
+     13 /v2/repository-provider/github/sign/<id>/<id>/<id>
+     12 /v2/user-from-token
+      8 /v2/github/installation
+      5 /v2/health
+      1 /v2/users/company/abcd
+      1 /v2/user-from-session
+      1 /v2/return-url/<uuid>
+      1 /v2/check-prepare-employee-signature
+      1 /v1/users/company/abcd
+```
+
+

utils/get_project.py renamed to utils/get_project_py.sh

@@ -3,14 +3,20 @@
 # Copyright The Linux Foundation and each contributor to LFX.
 # SPDX-License-Identifier: MIT
 
-# REGION=us-east-1|us-east-2 STAGE=dev DEBUG=1 DTFROM='3 days ago' DTTO='2 days ago' ./utils/search_aws_logs.sh 'error'
+# REGION=us-east-1|us-east-2 STAGE=dev DEBUG=1 DTFROM='3 days ago' DTTO='2 days ago' OUT=logs.json ./utils/search_aws_logs.sh 'error'
+# DEBUG=1 STAGE=dev REGION=us-east-1 DTFROM='10 days ago' DTTO='1 second ago' OUT=api-logs-dev.json ./utils/search_aws_logs.sh 'LG:api-request-path' && jq -r '.[].message' api-logs-dev.json | grep -o 'LG:api-request-path:[^[:space:]]*' | sed 's/^LG:api-request-path://' | sed -E 's/[0-9a-fA-F-]{36}/<uuid>/g' | sed -E 's/\b[0-9]{2,}\b/<id>/g' | sort | uniq -c | sort -nr
+# DEBUG=1 STAGE=prod REGION=us-east-1 NO_ECHO=1 DTFROM='10 days ago' DTTO='1 second ago' OUT=api-logs-prod.json ./utils/search_aws_logs.sh 'LG:api-request-path' && jq -r '.[].message' api-logs-prod.json | grep -o 'LG:api-request-path:[^[:space:]]*' | sed 's/^LG:api-request-path://' | sed -E 's/[0-9a-fA-F-]{36}/<uuid>/g' | sed -E ':a;s#/([0-9]{1,})(/|$)#/<id>\2#g;ta' | sort | uniq -c | sort -nr
 # To find distinct log groups: | jq -r 'map(.logGroupName) | unique | .[]'
 # in us-east-1 (mostly V1, V2 and V3):
 # To see specific log group: | jq 'map(select(.logGroupName == "/aws/lambda/cla-backend-dev-apiv1"))'
 # To filter out one log group: | jq 'map(select(.logGroupName != "/aws/lambda/cla-backend-dev-apiv2"))'
 # To filter out one log group: | jq 'map(select(.logGroupName != "cla-backend-dev-api-v3-lambda"))'
 # in us-east-2 (V4):
 # To filter out one log group: | jq 'map(select(.logGroupName != "cla-backend-go-api-v4-lambda"))'
+# To exclude some log groups: EXCL_LOGS="apiv1,apiv3"
+# To include only specific log groups INCL_LOGS="apiv1,apiv3"
+# All log groups in us-east-1: backend-dev-api-v3-lambda,backend-dev-apiv1,backend-dev-apiv2,backend-dev-authorizer,backend-dev-dynamo-events-events-lambda,backend-dev-dynamo-github-orgs-events-lambda,backend-dev-dynamo-projects-cla-groups-events-lambda,backend-dev-dynamo-projects-lambda,backend-dev-dynamo-repositories-events-lambda,backend-dev-dynamo-signatures-events-lambda,backend-dev-githubactivity,backend-dev-githubinstall,backend-dev-gitlab-repository-check-lambda,backend-dev-report-metrics-lambda,backend-dev-salesforceprojectbyID,backend-dev-salesforceprojects,backend-dev-save-metrics-lambda,backend-dev-user-event-handler-lambda,backend-dev-zip-builder-lambda,backend-dev-zip-builder-scheduler-lambda,backend-go-api,dev-stream-test-handler,dynamo-events-lambda,landing-page-dev-clientEdge,metrics-lamdba-test,prasanna-zip-builder,prasanna-zipbuilder-schedular,test-zipbuilder-lambda
+# All log groups in us-east-2: backend-go-api-v4-lambda
 
 if [ -z "${STAGE}" ]
 then
@@ -44,9 +50,43 @@ else
 fi
 
 mapfile -t log_groups_array < <(aws --region "${REGION}" --profile "lfproduct-${STAGE}" logs describe-log-groups --log-group-name-prefix "/aws/lambda/cla-" --query "logGroups[].logGroupName" | jq -r '.[]')
+
+IFS=',' read -ra INCL_LOGS_ARRAY <<< "${INCL_LOGS}"
+IFS=',' read -ra EXCL_LOGS_ARRAY <<< "${EXCL_LOGS}"
+
 results=()
 for log_group in "${log_groups_array[@]}"
 do
+  short_log_group="${log_group#/aws/lambda/cla-}"
+  # Skip if not in INCLUDE list (when it's set)
+  if [ ! -z "${INCL_LOGS}" ]
+  then
+    skip_included=true
+    for incl in "${INCL_LOGS_ARRAY[@]}"
+    do
+      if [[ "$short_log_group" == *"$incl"* ]]
+      then
+        skip_included=false
+        break
+      fi
+    done
+    if [ "$skip_included" = true ]
+    then
+      [ ! -z "$DEBUG" ] && echo "Skipping (not in INCLUDE): $log_group" >&2
+      continue
+    fi
+  fi
+
+  # Skip if in EXCLUDE list
+  for excl in "${EXCL_LOGS_ARRAY[@]}"
+  do
+    if [[ "$short_log_group" == *"$excl"* ]]
+    then
+      [ ! -z "$DEBUG" ] && echo "Skipping (in EXCLUDE): $log_group" >&2
+      continue 2
+    fi
+  done
+
   if [ ! -z "${DEBUG}" ]
   then
     echo "lookup log group '${log_group}': aws --region "${REGION}" --profile \"lfproduct-${STAGE}\" logs filter-log-events --log-group-name \"$log_group\" --start-time \"${DTFROM}\" --end-time \"${DTTO}\" --filter-pattern \"${search}\"" >&2
@@ -68,11 +108,18 @@ done
 
 jsons=""
 for log in "${results[@]}"; do
-    if [ -n "$jsons" ]
+    if [ ! -z "$jsons" ]
     then
         jsons+=$'\n'
     fi
     jsons+=$(echo "$log" | jq -r '.')
 done
 
-echo "$jsons" | jq -s 'sort_by(.dt) | reverse'
+if [ ! -z "${OUT}" ]
+then
+  echo "$jsons" | jq -s 'sort_by(.dt) | reverse' > "${OUT}"
+fi
+if [ -z "${NO_ECHO}" ]
+then
+  echo "$jsons" | jq -s 'sort_by(.dt) | reverse'
+fi