Bulk create custom targeting values for price targets

[Austinb]

I have noticed since the new api came out more issues with random connection issues with the DFP API when creating all of the price targeting values. I added a time.sleep() setting line 255 of add_new_prebid_partner.py but the issue seems to still happen randomly regardless of the sleep value used. I usually get an error related to the connection not responding from the API's side. I set the sleep to 0.1 without issues then that has issues. I bumped it to 0.5 and sometimes that has issues. I did 1.0 and had issues.

Has anyone else noticed any similar issues when creating the price targets? I have not seen any issues when creating the associations or any of the other calls. It seems like maybe the API has changed the requests per second or maybe the API is just having issues.

Usual error:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1346, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1346, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/path/dfp-prebid-setup/tasks/add_new_prebid_partner.py", line 435, in <module>
    main()
  File "/home/path/dfp-prebid-setup/tasks/add_new_prebid_partner.py", line 430, in main
    creative_template_id=creative_template_id
  File "/home/path/dfp-prebid-setup/tasks/add_new_prebid_partner.py", line 97, in setup_partner
    creative_template_id=creative_template_id)
  File "/home/path/dfp-prebid-setup/tasks/add_new_prebid_partner.py", line 235, in create_line_item_configs
    hb_pb_value_id = HBPBValueGetter.get_value_id(price_str)
  File "/home/path/dfp-prebid-setup/tasks/add_new_prebid_partner.py", line 183, in get_value_id
    val_id = self._create_value_and_return_id(value_name)
  File "/home/path/dfp-prebid-setup/tasks/add_new_prebid_partner.py", line 170, in _create_value_and_return_id
    self.key_id)
  File "/home/path/dfp-prebid-setup/dfp/create_custom_targeting.py", line 77, in create_targeting_value
    values_config)
  File "/usr/local/lib/python3.6/dist-packages/googleads/common.py", line 992, in MakeSoapRequest
    *packed_args, _soapheaders=soap_headers)['body']['rval']
  File "/usr/local/lib/python3.6/dist-packages/zeep/proxy.py", line 45, in __call__
    kwargs,
  File "/usr/local/lib/python3.6/dist-packages/zeep/wsdl/bindings/soap.py", line 122, in send
    response = client.transport.post_xml(options["address"], envelope, http_headers)
  File "/usr/local/lib/python3.6/dist-packages/zeep/transports.py", line 95, in post_xml
    return self.post(address, message, headers)
  File "/usr/local/lib/python3.6/dist-packages/zeep/transports.py", line 62, in post
    address, data=message, headers=headers, timeout=self.operation_timeout
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 567, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 490, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

[kmjennison]

Interesting. I haven't experienced this myself yet. How many prices are you creating when this happens? Curious if it's related to the total number or if it's just an occasional network issue.

[Austinb]

Sorry for the delayed response. We usually run batches of 400 pricing targets as part of a larger insert of orders into a GAM account. This is $0.10 to $40.00 in $0.10 increments. Unfortunately I do not think there is a way to batch insert these into GAM via api as far as I can tell. It only happens once but this part is where the GAM api is having issues or throttling or something. I tried different wait times but it seems to be random and not affect by the speed of the requests all the time.

[Austinb]

So I think this is more of an issue with changes to GAM's API side and them wanting you to batch insert as much as possible. The code currently does each item one at a time when sending them to the createCustomTargetingValues instead of batch inserting values. Changing this behavior would require a rewrite to the code since everything is done as it is iterated over. My suggestion is most things should be converted to batch inserts where possible. It is done currently in other places such as the line item associations.

[kmjennison]

Thanks for the info. Yeah, it would be better to pass an array of values to create_targeting_value in create_custom_targeting.py. This would require changes to the DFPValueIdGetter in add_new_prebid_partner.py.