Fetch all pages

Fetch all pages is a switch on the API Request and Shopify GraphQL steps. Turn it on and the step keeps asking for the next page until there are no more, then hands you every item in one list. You don't need a Loop, a page counter or a cursor of your own.

When to use it

Read every sale, product, customer or stock row from a Smooth POS list endpoint.
Read every Shopify order or product with a GraphQL query.
Stream a large list straight into a file with the File step, without holding it all in memory.

Turn it on

Add an API Request step and pick a list endpoint, like List Sales or Get Products, or add a Shopify GraphQL step. The switch only appears on endpoints that return a paged list.
Under Options, turn on Fetch all pages.
Fill in the fields below, and set a Response variable.

In the app	`data` field	What it does
Items path	`items_path`	Dot path to the list in each page, like `data.orders.nodes`. Leave it blank when the whole response is the list.
Explode array path	`explode`	Optional. Dot path of an array on each item, like `lines`. You get one item per element instead (for example, one row per sale line).
Keep rows with no lines	`explode_keep_empty`	With Explode array path, keeps one item for a parent whose array is empty or missing.
Max pages	`max_pages`	Default 20, up to 200.
Max items	`max_items`	Default 5,000, up to 50,000.
File handle	`file_handle`	Optional. The handle of a file you opened with a File step. Each page is written into the file, and `items` stays empty.
Cursor path	`cursor_path`	Shopify GraphQL only. Where the next cursor is, like `data.orders.pageInfo.endCursor`.
Has next path	`has_next_path`	Shopify GraphQL only. Where the "more pages" flag is, like `data.orders.pageInfo.hasNextPage`.
Cursor variable	`cursor_variable`	Shopify GraphQL only. The GraphQL variable that receives the cursor. Defaults to `cursor`.

For Shopify GraphQL, leaving the three cursor fields blank uses pageInfo.endCursor, pageInfo.hasNextPage and cursor.

What it saves

The response variable gets:

Field
`items`	Every item from every page, as one array (empty when you use a File handle).
`count`	How many items there are.
`pages`	How many pages were fetched.
`total`	The total the API reported, when it reports one.
`truncated`	`true` when Max pages or Max items stopped the step before the last page.
`next`	Where the next page would have started, when the step stopped early.

items is a real array, so you can use it straight away in a Transform Data step (Source orders.items), a Loop, or a script ($.orders.items.length).

🚧
Any bad page fails the step
With Fetch all pages on, Fail on bad response code is always on: if any page comes back with an error, the step fails and the job is retried.

Example 1: Alert on open Shopify orders

Trigger: Scheduled.

Shopify GraphQL: Fetch every order page, with Fetch all pages on.

GraphQL query:

query Orders($cursor: String) {
  orders(first: 100, after: $cursor, query: "created_at:>-1d") {
    nodes { id name displayFulfillmentStatus totalPriceSet { shopMoney { amount } } }
    pageInfo { hasNextPage endCursor }
  }
}

Items path: data.orders.nodes
Cursor path: data.orders.pageInfo.endCursor, Has next path: data.orders.pageInfo.hasNextPage
Response variable: orders

Transform Data: Keep the open orders, Source orders.items, one Filter operation: Field displayFulfillmentStatus, Equals, UNFULFILLED. Response variable open_orders.
If Condition: If any orders came back, Condition $.open_orders.length > 0.
Send Notification: Send a Slack message, under THEN, with Message {{ open_orders | size }} orders are still open.

Your query must accept the cursor variable ($cursor: String) and pass it to after:, and it must ask for pageInfo { hasNextPage endCursor }.

Example 2: Stream every order into a file

Trigger: Scheduled.

File: Open the export, Operation Open, Handle orders_csv, Format CSV, with your columns and destination (see File).
Shopify GraphQL: Fetch every order page, as in Example 1, with File handle orders_csv.
File: Close the export, Operation Close, Handle orders_csv, Response variable export.
If Condition: If the file has rows, Condition $.export.rows > 0, with a Send Notification step under THEN.

Tips

Raise the caps on purpose. If truncated is true, either raise Max pages and Max items or narrow the query (for example, by date).
Mind the job's time limit. A whole Glue job has 270 seconds. Very large lists are better split by date across several runs.
Use Explode for line-level rows. explode on lines turns 100 sales into one item per sale line, with the sale's fields on every row and a lines_index field giving the line's position.

← All Glue step guides