Functions can serve as a data source for a connector. When the connector is run, the function will be invoked.
Similar to Function transforms, the Function must have a string => string signature, meaning it will take a single string argument and return a single string. However, when using a function as a connector source, the strings passed into and returned by the function should be stringified JSON objects.
Formatting the Function
The string that should be passed in the function will be a stringified JSON object that has one key: pageToken. This is so that your function can support API pagination.
The first time the function is invoked by your connector, the pageToken value will be blank. If your function needs to be invoked more than once to fetch multiple pages of data, a nextPageToken value can be passed back into the function.
Returning Data
Once a function invocation is complete, it should return a stringified JSON object that has two keys: data and nextPageToken.
data is the JSON data returned by the function — it must be valid JSON structure in order to be processed by the connector.
nextPageToken is an optional value that should be set if there are more pages of data to be fetched. The value for nextPageToken is passed back to the function in the pageToken key as described above. If no nextPageToken is returned, the connector will assume there are no more pages of data to fetch and will not invoke the function again.
export async function myCustomDataSource(inputString: string): string {
const inputJson = JSON.parse(inputString)
const pageToken = inputJson["pageToken"]
let requestUrl = 'https://myrequesturl.com'
if (pageToken != null) {
requestUrl = `${requestUrl}&pagination_token=${pageToken}`
}
const response = await fetch(requestUrl).then(response => response.json())
const nextPageToken = response["meta"]["next_token"]
const outputString = JSON.stringify({"data": response, "nextPageToken": nextPageToken})
return outputString;
}