DENG: Taboola Requests Retry

[jsu]

Summary

1. Add tenacity.retry to requests.request()

Deployment Instructions

1. merge
2. add new release

[jsu]

Local CLI

With timeout set to 0.001 second. (Trigger timeout)

INFO:ingestor:approximate command line was: 

/OpenMail/etl/ingestor/ingest.py
	--insert
	run
	--provider-id
	4
	--report
	campaign_items_metadata_daily
	--data-ts=2020-04-26 00:00:00
	--account-id
	iac-shopping

INFO:ingestor:Running in test-mode
INFO:ingestor:Running data_ts 1/1: 2020-04-26 00:00:00
INFO:etl.ingestor.framework.worker:Task data is invalid.
INFO:etl.ingestor.framework.worker:Got task on step: download
INFO:etl.ingestor.framework.worker:Beginning download
INFO:botocore.credentials:Found credentials in environment variables.
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): ssm.us-west-2.amazonaws.com
INFO:pytaboola.client:Retrying pytaboola.client.TaboolaClient.execute in 1.0 seconds as it raised HTTPSConnectionPool(host='backstage.taboola.com', port=443): Max retries exceeded with url: /backstage/oauth/token (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7fdc2c44a588>, 'Connection to backstage.taboola.com timed out. (connect timeout=0.001)')).
INFO:pytaboola.client:Retrying pytaboola.client.TaboolaClient.execute in 2.0 seconds as it raised HTTPSConnectionPool(host='backstage.taboola.com', port=443): Max retries exceeded with url: /backstage/oauth/token (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7fdc2c44aac8>, 'Connection to backstage.taboola.com timed out. (connect timeout=0.001)')).
INFO:pytaboola.client:Retrying pytaboola.client.TaboolaClient.execute in 4.0 seconds as it raised HTTPSConnectionPool(host='backstage.taboola.com', port=443): Max retries exceeded with url: /backstage/oauth/token (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7fdc2c46d128>, 'Connection to backstage.taboola.com timed out. (connect timeout=0.001)')).

With timeout set to 3 seconds. (1 timed out then passed on retry)

INFO:ingestor:approximate command line was: 

/OpenMail/etl/ingestor/ingest.py
	--insert
	run
	--provider-id
	4
	--report
	campaign_items_metadata_daily
	--data-ts=2020-04-26 00:00:00
	--account-id
	iac-shopping

INFO:ingestor:Running in test-mode
INFO:ingestor:Running data_ts 1/1: 2020-04-26 00:00:00
INFO:etl.ingestor.framework.worker:Task data is invalid.
INFO:etl.ingestor.framework.worker:Got task on step: download
INFO:etl.ingestor.framework.worker:Beginning download
INFO:botocore.credentials:Found credentials in environment variables.
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): ssm.us-west-2.amazonaws.com
INFO:etl.ingestor.adapter.taboola:Getting campaigns for account iac-shopping
INFO:etl.ingestor.adapter.taboola:[0] Getting items for campaign delete
INFO:etl.ingestor.adapter.taboola:[1] Getting items for campaign shop-tab-wl25-us-m-f150-1
INFO:etl.ingestor.adapter.taboola:[2] Getting items for campaign shop-tab-wl25-us-m-silverado-1
INFO:etl.ingestor.adapter.taboola:[3] Getting items for campaign shop-tab-wl25-us-m-ram-1
INFO:etl.ingestor.adapter.taboola:[4] Getting items for campaign shop-tab-q36-us-m-eczema-1
INFO:etl.ingestor.adapter.taboola:[5] Getting items for campaign shop-tab-q36-us-m-psoarth-1
INFO:etl.ingestor.adapter.taboola:[6] Getting items for campaign shop-tab-wl25-us-m-bizcc-1
INFO:etl.ingestor.adapter.taboola:[7] Getting items for campaign shop-tab-wl25-us-m-zerointerestcc-1
INFO:etl.ingestor.adapter.taboola:[8] Getting items for campaign shop-tab-q36-us-m-zerointerestcc-1
INFO:pytaboola.client:Retrying pytaboola.client.TaboolaClient.execute in 1.0 seconds as it raised HTTPSConnectionPool(host='backstage.taboola.com', port=443): Read timed out. (read timeout=3).
INFO:etl.ingestor.adapter.taboola:[9] Getting items for campaign shop-tab-q36-us-m-bizcc-1
INFO:etl.ingestor.adapter.taboola:[10] Getting items for campaign shop-tab-q36-us-m-checking-1
INFO:etl.ingestor.adapter.taboola:[11] Getting items for campaign shop-tab-q36-us-m-anxidepress-1
INFO:etl.ingestor.adapter.taboola:[12] Getting items for campaign shop-tab-q36-us-m-crohns-1
INFO:etl.ingestor.adapter.taboola:[13] Getting items for campaign shop-tab-q36-us-m-lungc-1
INFO:etl.ingestor.adapter.taboola:[14] Getting items for campaign shop-tab-q36-us-m-rheumatoid-arth-1
INFO:etl.ingestor.framework.worker:Download took 13.30s
INFO:etl.ingestor.framework.worker:Persisting download step to s3
INFO:etl.ingestor.framework.worker:upload to s3 complete
INFO:etl.ingestor.framework.worker:Beginning extract
INFO:etl.ingestor.framework.worker:Beginning transform
INFO:etl.ingestor.framework.worker:transform + write took 0.01s
INFO:etl.ingestor.framework.worker:Persisting transform step to s3
INFO:etl.ingestor.framework.worker:upload to s3 complete
INFO:etl.ingestor.framework.worker:Beginning load
    
            SELECT account_id
            FROM account_metadata_scd
            WHERE provider_id = :provider_id
            AND manager_account_id is not NULL
            AND effective_to > current_timestamp
    
    with provider_id = 4
    SELECT t.* FROM ingestor.task_test t WHERE t.provider_id = :provider_id AND t.report = :report AND t.account_id = :account_id AND t.data_ts = :data_ts
    
    with account_id = iac-shopping
          data_ts = 2020-04-26 00:00:00
          provider_id = 4
          report = campaign_items_metadata_daily
    SELECT m.* FROM manager_account_mapping m JOIN account_metadata_scd a ON m.id = a.manager_account_id AND a.effective_to > current_timestamp WHERE a.account_id = :account_id AND m.provider_id = :provider_id AND m.is_active
    
    with account_id = iac-shopping
          provider_id = 4
    SELECT c.* FROM network_credential c JOIN manager_account_mapping_network_credentials m ON c.id = m.network_credential_id WHERE m.manager_account_mapping_id = :manager_account_id ORDER BY random()
    
    with manager_account_id = 54
    UPDATE ingestor.task_test t SET last_loaded = :ingest_datetime, is_valid = NULL WHERE t.id = :task_id
    
    with ingest_datetime = 2020-04-27 23:46:11
          task_id = 572556
Mon Apr 27 23:46:26 2020 database.py processed 3 queries on console during the current execution.
Mon Apr 27 23:46:26 2020 database.py processed 2 queries on metadb during the current execution.
Mon Apr 27 23:46:26 2020 Total queries processed: 5