Add traffic collection and slack messaging if those fail #9

pvk-developer · 2025-03-10T12:51:40Z

Resolves #5
CU-86b40hf4k

Workflow: https://github.com/datacebo/github-analytics/actions/runs/13764032556
The workflow failed because the token doesn't have access to the repo data. I'm not sure if we have to add it to the organization or something (it is the [email protected] account.

Results are stored in here and organized in folders, one per each sdv-dev repo. Then filenames are a timestamp. Here is SDV example

gsheni · 2025-03-10T15:10:46Z

.github/workflows/traffic_collection.yaml

+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install invoke
+        python -m pip install -e .[dev]


Suggested change

python -m pip install -e .[dev]

python -m pip install .[dev]

gsheni · 2025-03-10T15:14:00Z

github_analytics/github/traffic.py

+            repo (str):
+                The repository in the format "owner/repo".
+            endpoint (str):
+                The traffic API endpoint (e.g., "popular/referrers" or "popular/paths").


Let's list the values this could be.

Suggested change

The traffic API endpoint (e.g., "popular/referrers" or "popular/paths").

The traffic API endpoint (e.g., "popular/referrers", "popular/paths", "views", "clones").

github_analytics/github/traffic.py

gsheni · 2025-03-10T15:17:47Z

github_analytics/main.py

+        ouptut_folder (str):
+            Folder in which the metrics will be stored.
+    """
+    timestamp = datetime.datetime.now().strftime('%Y-%m-%d_%H-%M-%S')


Is it possible to save the timeframe for the data as another sheet? So the start and end timestamp (last 2 weeks)?

traffic_config.yaml

amontanez24

Looking good to me!

.github/workflows/traffic_collection.yaml

amontanez24 · 2025-03-10T22:17:49Z

github_analytics/drive.py

+    if parent_folder.startswith('gdrive://'):
+        parent_folder = parent_folder.replace('gdrive://', '')


minor: can we make 'gdrive://' a constant?

This file was updated 614e4b7

pvk-developer · 2025-03-11T12:42:52Z

Addressed @gsheni feedback, here are the results: https://docs.google.com/spreadsheets/d/1ggFsHydE7csoL95qJHRtR2LXQMTotYi07SkoH3lEpmE/edit?usp=sharing
Also added @amontanez24 's feedback in this same commit: a9bd174

gsheni

Will approve after renaming for readability is addressed. I want us to match the headers/title seen on the GitHub page:

gsheni · 2025-03-11T14:04:50Z

github_analytics/github/traffic.py

+        """
+        LOGGER.info(f'Fetching traffic referrers for {repo}.')
+        data = self._get_traffic_data(repo, 'popular/referrers')
+        df = pd.DataFrame(data, columns=['referrer', 'count', 'uniques'])


Let's match what we see on GitHub. These titles will make it easier for Kalyan and non-engineering people to understand the data.

Suggested change

df = pd.DataFrame(data, columns=['referrer', 'count', 'uniques'])

df = pd.DataFrame(data, columns=['site', 'views', 'unique_visitors'])

gsheni · 2025-03-11T14:05:51Z

github_analytics/github/traffic.py

+        """
+        LOGGER.info(f'Fetching traffic paths for {repo}.')
+        data = self._get_traffic_data(repo, 'popular/paths')
+        df = pd.DataFrame(data, columns=['path', 'title', 'count', 'uniques'])


Suggested change

df = pd.DataFrame(data, columns=['path', 'title', 'count', 'uniques'])

df = pd.DataFrame(data, columns=['content', 'title', 'views', 'unique_visitors'])

gsheni · 2025-03-11T14:06:26Z

github_analytics/github/traffic.py

+                    - `uniques`: Number of unique visitors.
+        """
+        data = self._get_traffic_data(repo, 'views')
+        return pd.DataFrame(data['views'], columns=['timestamp', 'count', 'uniques'])


Suggested change

return pd.DataFrame(data['views'], columns=['timestamp', 'count', 'uniques'])

return pd.DataFrame(data['views'], columns=['timestamp', 'views', 'unique_visitors'])

gsheni · 2025-03-11T14:06:38Z

github_analytics/github/traffic.py

+                    - `uniques`: Number of unique cloners.
+        """
+        data = self._get_traffic_data(repo, 'clones')
+        return pd.DataFrame(data['clones'], columns=['timestamp', 'count', 'uniques'])


Suggested change

return pd.DataFrame(data['clones'], columns=['timestamp', 'count', 'uniques'])

return pd.DataFrame(data['clones'], columns=['timestamp', 'clones', 'unique_cloners'])

gsheni · 2025-03-11T14:07:56Z

github_analytics/github/traffic.py

+            'Traffic Referrers': self.get_traffic_referrers(repo),
+            'Traffic Paths': self.get_traffic_paths(repo),
+            'Traffic Views': self.get_traffic_views(repo),
+            'Traffic Clones': self.get_traffic_clones(repo),


Suggested change

'Traffic Referrers': self.get_traffic_referrers(repo),

'Traffic Paths': self.get_traffic_paths(repo),

'Traffic Views': self.get_traffic_views(repo),

'Traffic Clones': self.get_traffic_clones(repo),

'Traffic Referring Sites': self.get_traffic_referrers(repo),

'Traffic Popular Content': self.get_traffic_paths(repo),

'Traffic Visitors': self.get_traffic_views(repo),

'Traffic Git Clones': self.get_traffic_clones(repo),

pvk-developer · 2025-03-11T15:21:35Z

Will approve after renaming for readability is addressed. I want us to match the headers/title seen on the GitHub page:

New format.

pvk-developer added 8 commits March 10, 2025 12:41

Add traffic collection and slack messaging if those fail

78f1c92

Fix lint

36011b6

Update weekly config

4a030e8

Test workflow

8291149

Update cancelled status

ddd8965

Fix github.job

7f03c2a

Use py313

0371756

Undo workflow changes

c129ee8

pvk-developer self-assigned this Mar 10, 2025

pvk-developer requested review from amontanez24 and gsheni March 10, 2025 12:51

gsheni requested changes Mar 10, 2025

View reviewed changes

amontanez24 reviewed Mar 10, 2025

View reviewed changes

Add clones, views and timeframe

a9bd174

pvk-developer requested review from amontanez24 and gsheni March 11, 2025 12:45

Make gdrive in drive.py constant

614e4b7

pvk-developer force-pushed the issue-5-save-traffic-from-sdv-repos branch from 444938b to 614e4b7 Compare March 11, 2025 12:46

gsheni requested changes Mar 11, 2025

View reviewed changes

gsheni approved these changes Mar 11, 2025

View reviewed changes

Update colum names and spreadsheets

22f04b4

pvk-developer force-pushed the issue-5-save-traffic-from-sdv-repos branch from 584c7ec to 22f04b4 Compare March 11, 2025 15:20

Fix lint

ce69353

pvk-developer merged commit 653234d into main Mar 11, 2025
1 check passed

pvk-developer deleted the issue-5-save-traffic-from-sdv-repos branch March 11, 2025 22:27

	The traffic API endpoint (e.g., "popular/referrers" or "popular/paths").
	The traffic API endpoint (e.g., "popular/referrers", "popular/paths", "views", "clones").

		if parent_folder.startswith('gdrive://'):
		parent_folder = parent_folder.replace('gdrive://', '')

	df = pd.DataFrame(data, columns=['referrer', 'count', 'uniques'])
	df = pd.DataFrame(data, columns=['site', 'views', 'unique_visitors'])

	df = pd.DataFrame(data, columns=['path', 'title', 'count', 'uniques'])
	df = pd.DataFrame(data, columns=['content', 'title', 'views', 'unique_visitors'])

	return pd.DataFrame(data['views'], columns=['timestamp', 'count', 'uniques'])
	return pd.DataFrame(data['views'], columns=['timestamp', 'views', 'unique_visitors'])

	return pd.DataFrame(data['clones'], columns=['timestamp', 'count', 'uniques'])
	return pd.DataFrame(data['clones'], columns=['timestamp', 'clones', 'unique_cloners'])

Add traffic collection and slack messaging if those fail #9

Add traffic collection and slack messaging if those fail #9

Uh oh!

Conversation

pvk-developer commented Mar 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

amontanez24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pvk-developer Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pvk-developer commented Mar 11, 2025

Uh oh!

gsheni left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pvk-developer commented Mar 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pvk-developer Mar 11, 2025 •

edited

Loading

gsheni left a comment •

edited

Loading