You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+40-13Lines changed: 40 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,21 @@
1
-
# Megalista
1
+
# MDS - Marketing Data Sync
2
2
3
-
Sample integration code for onboarding offline/CRM data from BigQuery as custom audiences or offline conversions in Google Ads, Google Analytics 360, Google Display & Video 360 and Google Campaign Manager.
3
+
Solution based on the [Google Megalista project](https://github.com/google/megalista).
4
4
5
-
**Disclaimer:** This is not an officially supported Google product.
Sample integration code for onboarding offline/CRM data from BigQuery as custom audiences or offline conversions in Google Ads, Google Analytics 360, Google Display & Video 360, Google Campaign Manager and Facebook Ads.
6
19
7
20
## Supported integrations
8
21
-**Google Ads**
@@ -25,7 +38,7 @@ Sample integration code for onboarding offline/CRM data from BigQuery as custom
25
38
- S2S Offline events API (conversion upload), to be used for audience creation and in-app events with Google Ads and DV360 [[details]](https://support.appsflyer.com/hc/en-us/articles/207034486-API-de-eventos-de-servidor-para-servidor-S2S-mobile-para-mobile)
26
39
27
40
## How does it work
28
-
Megalista was design to separate the configuration of conversion/audience upload rules from the engine, giving more freedom for non-technical teams (i.e. Media and Business Inteligence) to setup multiple upload rules on their own.
41
+
MDS was design to separate the configuration of conversion/audience upload rules from the engine, giving more freedom for non-technical teams (i.e. Media and Business Inteligence) to setup multiple upload rules on their own.
29
42
30
43
The solution consists in #1 a Google Spreadsheet (template) in which all rules are defined by mapping a data source (BigQuery Table) to a destination (data upload endpoint) and #2, an apache beam workflow running on Google Dataflow, scheduled to upload the data in batch mode.
31
44
@@ -46,7 +59,7 @@ The solution consists in #1 a Google Spreadsheet (template) in which all rules a
46
59
-**Google Cloud SDK**
47
60
48
61
### Access Requirements
49
-
Those are the minimum roles necessary to deploy Megalista:
62
+
Those are the minimum roles necessary to deploy MDS:
50
63
- OAuth Config Editor
51
64
- BigQuery User
52
65
- BigQuery Job User
@@ -81,23 +94,23 @@ In order to create it, follow these steps:
81
94
- On the **OAuth Consent Screen** and configure an *Application name*
82
95
- Then, go to the **Credentials** and create an *OAuth client Id* with Application type set as *Desktop App*
83
96
- This will generate a *Client Id* and a *Client secret*
84
-
- Run the **generate_megalist_token.sh** script in this folder providing these two values and follow the instructions
- This will generate the *Access Token* and the *Refresh token*
87
100
88
101
### Creating a bucket on Cloud Storage
89
102
This bucket will hold the deployed code for this solution. To create it, navigate to the *Storage* link on the top-left menu on GCP and click on *Create bucket*. You can use Regional location and Standard data type for this bucket.
90
103
91
-
## Running Megalista
104
+
## Running MDS
92
105
93
106
We recommend first running it locally and make sure that everything works.
94
107
Make some sample tables on BigQuery for one of the uploaders and make sure that the data is getting correctly to the destination.
95
108
After that is done, upload the Dataflow template to GCP and try running it manually via the UI to make sure it works.
96
-
Lastly, configure the Cloud Scheduler to run Megalista in the frequency desired and you'll have a fully functional data integration pipeline.
109
+
Lastly, configure the Cloud Scheduler to run MDS in the frequency desired and you'll have a fully functional data integration pipeline.
97
110
98
111
### Running locally
99
112
```bash
100
-
python3 megalist_dataflow/main.py \
113
+
python3 mds_dataflow/main.py \
101
114
--runner DirectRunner \
102
115
--developer_token ${GOOGLE_ADS_DEVELOPER_TOKEN} \
103
116
--setup_sheet_id ${CONFIGURATION_SHEET_ID} \
@@ -119,15 +132,15 @@ To execute the pipeline, use the following steps:
119
132
- Go to **Dataflow** on GCP console
120
133
- Click on *Create job from template*
121
134
- On the template selection dropdown, select *Custom template*
122
-
- Find the *megalist* file on the bucket you've created, on the templates folder
135
+
- Find the *mds* file on the bucket you've created, on the templates folder
123
136
- Fill in the parameters required and execute
124
137
125
138
### Scheduling pipeline
126
139
To schedule daily/hourly runs, go to **Cloud Scheduler**:
127
140
- Click on *create job*
128
141
- Add a name and frequency as desired
129
142
- For *target* set as HTTP
130
-
- Configure a *POST* for url: https://dataflow.googleapis.com/v1b3/projects/${YOUR_PROJECT_ID}/locations/${LOCATION}/templates:launch?gcsPath=gs://${BUCKET_NAME}/templates/megalist, replacing the params with the actual values
143
+
- Configure a *POST* for url: https://dataflow.googleapis.com/v1b3/projects/${YOUR_PROJECT_ID}/locations/${LOCATION}/templates:launch?gcsPath=gs://${BUCKET_NAME}/templates/mds, replacing the params with the actual values
131
144
- For a sample on the *body* of the request, check **cloud_config/scheduler.json**
@@ -142,4 +155,18 @@ It's recommended to create a new Service Account to be used with the Cloud Sched
142
155
143
156
144
157
## Usage
145
-
Every upload method expects as source a BigQuery data with specific fields, in addition to specific configuration metadata. For details on how to setup your upload routines, refer to the [Megalista Wiki](https://github.com/google/megalista/wiki) or the [Megalista user guide](https://github.com/google/megalista/blob/main/documentation/Megalista%20-%20Technical%20User%20Guide%20-%20EXTERNAL.pdf).
158
+
Every upload method expects as source a BigQuery data with specific fields, in addition to specific configuration metadata. For details on how to setup your upload routines, refer to the [MDS Wiki](https://github.com/dp6/marketing-data-sync/wiki) or the [MDS user guide](https://github.com/dp6/marketing-data-sync/blob/main/documentation/mds%20-%20Technical%20User%20Guide%20-%20EXTERNAL.pdf).
159
+
160
+
### Mandatory requirements
161
+
162
+
Only contributions that meet the following requirements will be accepted:
0 commit comments