Pull from API

Any data that can be retrieved via an API (formatted as JSON) can be extracted and loaded into a connector. This is referred to as the Pull from API connector source.

Building a connector using an API as the data source requires familiarity with the source’s API documentation. This is a similar process to using a Native Source connector, but instead, the connection will be configured manually.

Setting Up a Pull from API Connector

Step 1: Create the Connector

1. Navigate to Knowledge Graph > Entities and click + Add Data.
2. Click Build Connector.
3. Select Pull from API as the connector source.
4. Select the entity type you would like to create or update with this connector.

Step 2: Configure API Details

Request URL

In this field, you will enter the GET request URL. A GET request is used to request data from an API.

For example, to retrieve videos from YouTube the GET URL for the YouTube API is https://www.googleapis.com/youtube/v3/search.

You can usually find the GET URL from the developer documentation. For YouTube, we found that information here: https://developers.google.com/youtube/v3/docs/search/list

Authentication Method

Most APIs require authentication in order to access their data. You will select the authentication method that is supported by the API you are trying to pull data from. Once you select the relevant option you will see the additional fields needed to configure that authentication method. For example, if you select API Key you will enter a name for the key and then your alphanumeric API key.

If you are using the YouTube API, you will need to generate an API Key to give you access. You can see the instructions on how to do so in YouTube’s YouTube Data API Overview article, or follow these instructions:

1. Log into the Google Developer Console  .
2. Create a new project by clicking Create New Project or by clicking on the name of the existing project you have open and clicking NEW PROJECT.
3. Once you’re in your new project, click + ENABLE APIS AND SERVICES.
4. In the API library, navigate to YouTube Data API v3.
5. Click ENABLE.
6. Then click CREATE CREDENTIALS and select the relevant settings.
7. You will then see a page display with your API key.

You can create a sample call by entering https://www.googleapis.com/youtube/v3/videos?key=YOUR_API_KEY into your browser.

You can also see a sample call in the Sample partial request section of Youtube’s Data API Overview article.

Query Parameters

This is where you enter specific query parameters that you need to make a successful call. You can add any number of key-value pairs that we will append to your GET request as query parameters.

For YouTube this will include the following:

• part - this parameter is required for the Search endpoint and must be set to snippet. This is discussed in the YouTube API documentation under “Parameters” here  .
• type - this parameter specifies the resource type we would like to filter the results to. For our example, we will set this to video to make sure our response only includes information about video resources.
• maxResults - this parameter specifies the number of videos you want returned per page. It is optional and will default to 5 if not specified, but let’s set it to 50. This way our preview data table in later steps will include 50 records instead of 5 (since we only show one response at a time in the preview table!)
• channelId - this parameter specifies the channel we would like to filter the results to. In the case of this example, the YouTube Channel ID for the Yext YouTube channel is UCBGyMZqAKM1FDAAEOR3s8Ag. This can also be located in the URL for the YouTube channel:

Headers

This is where you enter specific HTTP request headers that you need to make a successful call. You can add any number of key, value pairs that we will set as headers in your GET request.

For the YouTube Connector example, we actually do not need to set any specific Headers so we will skip this step and leave it blank!

Pagination Methods

Most API List endpoints support a pagination, meaning they will return the data you requested in multiple pages or responses. The API Documentation will typically let you know which pagination method the API uses and how you can fetch the additional pages.

The Youtube API’s Search endpoint uses cursor pagination, as discussed here.

When multiple pages exist, the nextPageToken property will be included in the API response and includes a token that lets us know that another page of results exists. This is called the cursor. We will set nextPageToken in the “Cursor” field so that we know to look for the cursor in the nextPageToken property of each response from the YouTube API.

Sometimes the cursor is contained in the response header, but in this case, it is contained in the response body so we will set the “Detect Cursor In” field to Response Body.

A cursor could be one of the following types: a token, a relative URL or a full URL. If the cursor is a token, it will be passed in the next API request using a query parameter so we must also know the parameter’s key (the “Page Key”). In this case, the YouTube API cursor is a token, and the key used to send that token in the next request is pageToken. If the cursor is a full URL or relative URL, you do not need to specify a Page Key!

Lastly, you can optionally specify a Max Page Count. In this example, we want to fetch all of the videos from the Yext channel, so we will leave this blank.

JSON Type

Next you need to specify if the response will contain one entity or many entities. When you look at the sample response you will be able to see this. If the response contains a list of entities you will need to add the JMESPath Syntax for that entity container so we can identify and pull the nested data for each.

Assuming you filled out all of the details above, you can run a test call by clicking Pull at the top of the page. Then click on the View Raw Response button so you can view all the video data and confirm that your pull request was successful. You can also see that the YouTube API response contains multiple videos, so we will select “Many Entities” as our JSON Type.

As you can see in your Raw Response, video objects are nested under the items section of the response, so enter items as the Entity Container.

Step 3: Specify Selectors

Selectors define which fields to extract from the API response and how to label them in the connector. 

We recommend looking at the sample response, as a reminder you can do this by clicking on the View Raw Response button at the top of the page. If you open this in a new tab you can easily see the components of the response. From there, you can pull the field names.

1. Click + Add Selector to get started.
2. Enter the Header — this will be the column header in your data preview.
3. Then, enter the JMES Path for the specific element you want to extract from your API response.
4. Click Save to add one selector, or Save & Add Another to add multiple.

For our Connector, we want to pull in: Video ID, Name, Description, Thumbnail URL, Published At, and Channel ID.

Below is a screenshot of what your sample response should look like. We have highlighted the relevant pieces of data we want to extract so you can understand the JMES Path expression for each.

• Video ID: id.videoId
• Name: snippet.title
• Description: snippet.description
• Thumbnail URL: snippet.thumbnails.default.url
• Published At: snippet.publishedAt
• Channel ID: snippet.channelId

Once you have added all of your relevant selectors, you should see a data table with some sample data. Click Continue to save these selectors and move on to the next step.

Step 4: Add Transforms

Optionally add transforms to clean or manipulate data before it is mapped to fields in Yext. For a full list of available transforms, see Connector Transforms.

In the Video Titles and Descriptions, YouTube returns some HTML character references instead of actual characters (e.g., & instead of “&“). We will add a “Find & Replace” transform to check for these HTML character references and swap them out with the corresponding characters.

For replacing & with “&”, the transform would look like this:

We will also add transforms to find and replace the following:

• ' with a single quote (‘)
• " with a double quote (“)

Step 5: Map Data

Map connector columns to fields on your selected entity type. For details on mapping options, field-type-specific behavior, and validation, see Connector Field Mapping.

Step 6: Managing Your Connector

Schedule: Set a run frequency from the connector settings — None (manual only), Daily, Weekly, or Monthly. When setting a schedule, you can also configure a Run Mode.

Trigger via API: You can trigger a run programmatically instead of (or in addition to) using the UI or a schedule:

1. Create an app in your Developer Account with write permission on the connectors endpoint.
2. POST a request to /v2/accounts/me/connectors/{connectorId}/trigger. Note that the POST request body will be empty, triggering a new run via the /trigger endpoint will start a run exactly as clicking the “Run Now” button in the UI does. Once you trigger a new run, the Connector will fetch data by making an API GET request according to its settings.

Request Details

Method

Designate whether the request should be a GET or POST request. POST requests are generally associated with GraphQL APIs.

Request URL

Provide the URL path to the API. It is not necessary to specify any query parameters, pagination settings, or authentication settings within the URL. If they are specified in their designated sections, the system will correctly construct the final API request URL.

Authentication Method

Select the method of authentication that the API supports from the following options:

• API Key: The API key will be added as a query parameter or header parameter, based on what’s specified.
• Bearer Token: The provided token will be added as an Authorization header, prepended with “Bearer.”
• Basic Authentication: Provide a username and password, which will then be Base 64 encoded by our system and added as an authorization header, prepended with “Bearer.”
• OAuth: If OAuth is the method of authentication, you’ll need to specify a linked account.  The Linked Account system manages all tokens and refresh tokens. If the provider for your API is not a built-in provider, follow the steps in the Create Custom OAuth Providers help article to create a custom OAuth provider and link your account.

Query Parameters

Specify any query parameters that should be appended to the final API request. Do not include authorization query parameters or pagination query parameters, as those should be provided in their respective sections.

As an example, if the following key value pair is provided as a query parameter:

• Key: v
• Value: 20240101

And if the request URL is https://www.myapi.com/test, the system will construct the following request: https://www.myapi.com/test?v=20230101

Headers

Specify any key-value pairs that should be added as additional headers to the API request. By default, the system adds:

• Content Type: application/json
• User Agent: YextConnector

Max Requests (Optional)

Specify a rate limit for the connector to adhere to. Provide the maximum number of requests the system should make to the API in a given unit of time, to ensure the rate limit is not exceeded.

Pagination

The Pull from API source accepts four pagination options:

1. Cursor
2. Link Header
3. Offset
4. Page-Based

Refer to the API documentation of your chosen API source to determine which method(s) may be supported.

For any of the methods, if Max Pages is provided, the connector will respect the provided value and not paginate beyond the maximum number of pages specified.

Cursor

APIs that utilize cursor pagination (such as the Yext APIs) will return a token or URL as part of the response body or headers, which can then be referenced as a query parameter in the next API call to fetch the next page of results.

In order for the connector to correctly paginate, it needs to know:

1. Cursor type: Is the cursor a token (i.e., a random string of characters), or is a full URL or relative URL returned?
2. Location: Is the cursor in the response body or in the header?
3. Cursor property: What property contains the value for the cursor?

   • For example, if the cursor is in the response body, the path to property with URL may be response.nextPageUrl for an API response like the following:

     { 
     response: {
     nextPageUrl: "https://myapi.com?nextpagetoken123"
           }
     }

     If the cursor type is token, provide the page key that should be passed as a query parameter in each API request to contain the value for the token provided in the previous page’s response. For example, the first page of the API response might look like this:

     { 
     response: {
     nextPageToken: hKLFJoin89041JKLg8hs9fjkdlsaH789HJ
           }
     }

     To paginate, the next API request may need to look like this: https://myapi.com?pageToken=hKLFJoin89041JKLg8hs9fjkdlsaH789HJ

In this case, the page key would be pageToken, as this query param contains the value of the token provided in the previous request, for the next page.

Link Header

APIs that use link header pagination have a header property link that contains direct links to other pages in the API response, such as to the next page, last page, previous page, and/or first page, as specified by the value in the relationship (rel) label.

For example, the header might look like the following:

Link: <https://myapi.com?page=15>; rel="next",
  <https://myapi.com?page=100>; rel="last",
  <https://myapi.com?page=1>; rel="first",
  <https://myapi.com?page=13>; rel="prev"

The connector can paginate by getting access to the URL for the next page, by specifying the following two settings:

1. Link type: Specify whether the link that is provided is the full URL to the next page, or just the relative URL (in the example above, the full URL is provided).
2. Relationship label: Specify the value for rel that contains the link to the next page (in the example above, this is next).

Offset

Offset pagination relies on passing an offset parameter, to specify the offset of the item to return from the first item in the list (or, the number of items to skip when fetching the next set of results). A limit parameter is required as well, to signify the maximum number of items to return in the response.

For example, if each page should contain 10 items, and you want to paginate to retrieve items 11-20, the resulting API request may look like this: https://www.myapi.com?limit=10&offset=10

In order for the connector to correctly paginate, it needs to know:

1. What is the offset key? (In the above example, this is offset)
2. What is the value for the first item (what is the offset start value)?
   • This is typically 0 or 1, but refer to the API documentation to confirm.
3. What is the limit key to contain the maximum entries per page? (In the above example, this is limit)
4. How many entries should be returned per page (what is the limit value)?
   • In the above example, the value is 10. Refer to your API documentation to learn what’s supported for your API.
5. What should tell the connector that the final page or final item has been reached?

There are two ways to signal to the connector that the final page has been reached:

1. Total pages key
   • If in the header: What is the key of that header containing the value? (e.g., API header contains X-TOTAL-PAGES: 10)
   • If in the response body: What is the JMESpath to the node containing the value? (e.g., for the following API response body, the path is response.totalPages)
2. Total items key
   • If in the header: What is the key of that header containing the value? (e.g., API header contains X-TOTAL-ITEMS: 10)
   • If in the response body: What is the JMESpath to the node containing the value? (e.g., for the following API response body, the path is response.totalItems)

Example response:

{ 
	response: {
		items: {
[
					item1data: {}
					...
				]
			},
		totalPages: 10
		totalItems: 100
          }
}

Page-Based

Page-based pagination relies on passing a query parameter for the page of the API response to return.

For example, the resulting API request may look like: https://www.myapi.com?pageNumber=3

For the connector to correctly paginate, it needs to know:

1. What is the format of the query param (what is the page key)?
   • In the above example, this is pageNumber.
2. What is the value of the query param for the first page (what is the start value)?
   • This is typically 0 or 1, but refer to the API documentation to confirm.
3. What should tell the connector that the final page has been reached?

There are two ways to signal to the connector that the final page has been reached:

1. Total pages key
   • If in the header: What is the key of that header containing the value? (e.g., API header contains X-TOTAL-PAGES: 10)
   • If in the response body: What is the JMESpath to the node containing the value? (e.g., for the following API response body, the path is response.totalPages)

   • Example response body:

     { 
     response: {
     items: {
             [
                 item1data: {}
                 ...
             ]
            },
     totalPages: 10
           }
     }

2. A key that contains the total number of entries returned per page (whereby if the connector reaches a page with fewer items, the final page is detected)
   • Limit key: The JMESpath to the key in the response body containing the value for the maximum number of entries returned in a single page
   • Limit value: (optional) provide a value for the limit key that is different from that of the default of your API source, which will be provided as a query parameter in each API call. For example, to request only 10 items per page, each API request may contain the following param: https://www.myapi.com?per_page=10

   • Example response body (where the limit key is per_page and the limit value is 10 items per page)

     { 
     response: {
     items: {
             [
                 item1data: {}
                 ...
             ]
            },
     per_page: 10
           }
     }

Limitations

The following pagination methods are not supported for a connector using the pull from API source:

1. LQS pagination
2. Pagination that requires the pagination settings to be passed in the response body (this is very likely for any GraphQL use cases)

You will need to use the function source to properly connect to these APIs and paginate.

If your API uses OAuth, but the grant type is not “Authorization Code” or “Client Credentials,” you will also need to use the function connector source.