OAuth not working during ingestion

[Roznoshchik]

About You:
Name: Rostislav
Company: Coursemojo

We have a bunch of data that's in dynamo db that we need to get into tableau.
It seemed that it would be quite simple to build a web connector to ingest that data. But it's been a painful process.

We have our own internal oauth server - when connecting, the application gets the access token and the refresh token. And is able to make requests to the api which returns the data.

But some of our tables are quite large and the access token expires before the data can be fetched fully. At that point, I would expect the sdk to use the refresh token to get a new access token and retry the request to continue ingesting the data.

Instead, the whole thing breaks due to an unexpected 401 error and none of the data that it had already ingested sticks around.

Am i misunderstanding how the whole oauth flow is supposed to work?

I can't find any mention in the documentation either about what I need to do to get it to refresh either. Is there something special I need to put into my fetcher class?

Also, not seeing these console logs anywhere either ... where do they appear?

import { Fetcher, FetchUtils, FetchOptions, DataRow, getOAuthHeader } from '@tableau/taco-toolkit/handlers'


export default class DataFetcher extends Fetcher {
  async *fetch({ secrets }: FetchOptions) {
    const baseUrl = 'https://website.com/data/events'
    const headers = getOAuthHeader(secrets)
     
    let count = 0
    let hasMoreData = true
    let startAfterKey = null

    while (hasMoreData && count <= 1000) {
      let fetchUrl = baseUrl
      if (startAfterKey) {
        const encodedStartAfterKey = encodeURIComponent(JSON.stringify(startAfterKey));
        fetchUrl = `${baseUrl}?start_after_key=${encodedStartAfterKey}`;
      }

      try {
        console.log("GETTING RESPONSE")
        const response = await FetchUtils.fetchJson(fetchUrl, { headers })
        const rows: DataRow[] = response.items;
        count += rows.length
        yield await FetchUtils.ingestDataRows(rows)

        startAfterKey = response.pagination.start_after_key
        hasMoreData = !!startAfterKey
      } catch (error) {
        console.error('Error in fetch:', error)
        throw error
      }
    }
  }
}

[Roznoshchik]

function fetchJson(networkAdapter, url, options) {
    return __awaiter(this, void 0, void 0, function () {
        var fetchParams, headers, response;
        return __generator(this, function (_a) {
            switch (_a.label) {
                case 0:
                    fetchParams = getFetchParams(url, options);
                    headers = fetchParams.options.headers;
                    if (headers === undefined) {
                        fetchParams.options.headers = headers = {};
                    }
                    headers['Accept'] = 'application/json';
                    return [4 /*yield*/, networkAdapter.fetch(fetchParams.urlWithQuery, fetchParams.options)];
                case 1:
                    response = _a.sent();
                    if (!response.ok || response.body === null) {
                        throw new Error("fetchJson: unexpected response. [status]: ".concat(response.status, ", [statusText]: ").concat(response.statusText));
                    }
                    return [4 /*yield*/, response.json()];
                case 2: return [2 /*return*/, _a.sent()];
            }
        });
    });
}
exports.fetchJson = fetchJson;

here is where the error is being thrown ... and nothing is catching it to refresh.

Is there a method to call or something I can do to get tableau to trigger the refresh?

[Roznoshchik]

{"ts":"2024-08-05T15:11:25.528Z","pid":"32509","sev":"info","sourceName":"Extractor.c4ca6874-bed5-4bad-a0c8-7f694f220779","v":"Count of rows ingested: 2"}
{"ts":"2024-08-05T15:11:26.991Z","pid":"32509","sev":"info","sourceName":"Extractor.c4ca6874-bed5-4bad-a0c8-7f694f220779","v":"Count of rows ingested: 2"}
{"ts":"2024-08-05T15:11:27.950Z","pid":"32509","sev":"error","sourceName":"Extractor.c4ca6874-bed5-4bad-a0c8-7f694f220779","v":"Error: fetchJson: unexpected response. [status]: 401, [statusText]: Unauthorized     at /snapshot/extensible-protocol-server/dist/eps/sandbox/sdk/fetcher-utils/fetch-utils.js     at step (/snapshot/extensible-protocol-server/dist/eps/sandbox/sdk/fetcher-utils/fetch-utils.js)     at Object.next (/snapshot/extensible-protocol-server/dist/eps/sandbox/sdk/fetcher-utils/fetch-utils.js)     at fulfilled (/snapshot/extensible-protocol-server/dist/eps/sandbox/sdk/fetcher-utils/fetch-utils.js)     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"}
{"ts":"2024-08-05T15:11:27.950Z","pid":"32509","sev":"error","sourceName":"Extractor.c4ca6874-bed5-4bad-a0c8-7f694f220779","v":"{ 'Status Code': 'UNHEALTHY', 'Error Source': 'client', 'EPS Error Code': 'P6WNIRTU', 'Error Message': 'Failure in sandbox pipeline. DataFetcher.fetch failed. fetchJson: unexpected response. [status]: 401, [statusText]: Unauthorized' }"}
{"ts":"2024-08-05T15:11:28.171Z","pid":"32509","sev":"info","sourceName":"Extractor.c4ca6874-bed5-4bad-a0c8-7f694f220779","v":"{ listeningServer: 'ExtractorApi', method: 'GET', type: 'received request', url: '/tables', headers: { host: 'localhost:9092', 'accept-encoding': 'gzip, deflate', 'user-agent': 'Tableau Desktop 20242.24.0711.1636; pro; libcurl-client; 64-bit; en_US; Mac OS X 14.5;', 'eps-instance-id': '8e418e5f-3f74-43ab-8973-9f38b86482aa', 'connection-id': 'coursemojo_data_router_v4.0i0h72407nsk5e188z69t1j5esua', 'extractor-id': 'c4ca6874-bed5-4bad-a0c8-7f694f220779', 'caller-id': 'J7yq/iY7EdLKbMidvZ98qD', accept: 'application/json', 'content-type': 'application/json', connection: '********' } }"}
{"ts":"2024-08-05T15:11:28.172Z","pid":"32509","sev":"debug","sourceName":"Extractor.c4ca6874-bed5-4bad-a0c8-7f694f220779","v":"Failure in sandbox pipeline. DataFetcher.fetch failed. fetchJson: unexpected response. [status]: 401, [statusText]: Unauthorized"}