Search API Connector Documentation

Print

Filter Fields (JMESPath)

By default, API Connector displays the entire response returned by any API request. However, there may be cases where you would like to query for specific fields or values in order to limit the data that gets returned or fix columns in place. API Connector provides this functionality through the JMESPath query syntax, which enables querying JSON APIs for specific fields and values.

This is a paid feature, please install API Connector for a free trial or upgrade to access.

Contents

Why Filter Responses?

  • avoid running up against cell limits in Sheets (currently 5 million cells maximum)
  • improve Sheets performance by reducing data size
  • simplify reporting by requesting only the fields you need
  • change the report structure, for example to convert columns to rows
  • enable consistent column order when an API shifts response fields between pulls

Before You Begin

If you haven’t already, click here to install the API Connector add-on from the Google Marketplace.

Get Your JSON

The first step of working with JMESPath is to view the JSON output of your API request.

To view the raw JSON, set up and save your request, and then click Output options > Preview API response. This will display the JSON response in a modal where you can copy or download it.

This article contains more information on previewing API responses: https://mixedanalytics.com/knowledge-base/preview-api-response/

Quick JSON Overview

If you already know about JSON, you can skip right past this section. JSON is a data format made up of objects and arrays, and we have to identify these structures to create JMESPath expressions.

  • Objects are unordered collections of name/value pairs. They start and end with curly braces.
{"wrapperType": "track",
"kind": "song",
"artistId": 657515}
  • Arrays are ordered lists of values. They start and end with square brackets. Arrays can contain lists of objects, like this:
"results": [
{ "artistName": "Radiohead",
  "collectionName": "In Rainbows",
  "trackName": "Weird Fishes / Arpeggi"},
{ "artistName": "Radiohead",
  "collectionName": "In Rainbows",
  "trackName": "15 Step"}
]

If you aren’t used to JSON yet, all you really need to know for now is if you see [] it’s an array, and if you see {} it’s an object.

General Syntax

The full JMESPath specification is a powerful but complex query language, so this article will focus only on the most common use case for API Connector: choosing which fields to print into your spreadsheet.

When we choose those fields, we need to identify the exact path to their location, where paths to values in arrays take the format array_name[].field and paths to values in objects take the format object_name.field.

The general syntax for setting up a JMESPath expression is like this, where we first provide a new key (name) for the field and then provide the path to that field:

array_name[].{new_key1:path1, new_key2:path2}

Examples are right below.

Example 1: Query for Specific Fields

In this example, we’ll request the collectionName and trackName fields from an example JSON response.

{
    "resultCount": 2,
    "results": [
        {
            "wrapperType": "track",
            "kind": "song",
            "artistId": 657515,
            "collectionId": 1109714933,
            "trackId": 1109715168,
            "artistName": "Radiohead",
            "collectionName": "In Rainbows",
            "trackName": "Weird Fishes / Arpeggi"
        }
    ]
}

By checking the JSON, we can see that trackName and collectionName are located inside the results array, such that our JMESPath expression looks like this. We’re keeping the key names the same:

results[].{trackName:trackName,collectionName:collectionName}

Enter this into API Connector like this:
jmespath-img2

This will result in the following output:
jmespath-img3

Example 2: Query Nested Fields

In some cases, values may be nested inside other objects and arrays. As mentioned above we need to recognize the data structure to identify the path. In this next example, β€œdata” is an array, while β€œbilling_details” is an object.

{  "data": [{
	"id": "1234567",
	"object": "charge",
	"amount": 100,
	"billing_details": {
		"email": "apple@email.com",
		"name": "Apple Appleby"
		}
	}]
}

To retrieve the amount and the email address, we’d set our query like this, since fields nested within objects are identified with a dot:

data[].{amount:amount,billing_email:billing_details.email}

Fields nested within arrays are accessed with [], so if you instead had, say, a product ID nested within a “products” array, you’d create an expression like this:

orders[].{order_id:order_id, product_id:products[].id}

Example 3: Query Two or More Top-Level Elements

I’m giving this its own section because it took me forever to figure out — and then it turned out the answer is very simple.

You may come across some JSON like this, where you want elements from two different top-level objects or arrays. For example, in this example JSON block you may want to get the number of records from the pagination object as well as date information from within the data array.

{    
  "pagination": {
        "limit": 100,
        "offset": 0,
        "count": 100,
        "total": 5000
    },
    "data": [
        {
            "date": "2020-06-08",
            "status": "scheduled",
            "timezone": "America/New_York"
    }]
}

To grab elements from both the pagination object and the data array at the same time, either of the following structures will work. You may find that one or the other provides a more convenient output for your data set.

{pagination_total:pagination.total,data:data[].{date:date,timezone:timezone}}
[{pagination_total:pagination.total},data[].{date:date,timezone:timezone}]

Tips

  • If you are working with deeply nested or complicated JSON, it might be hard to visually recognize the path to your desired fields. In those cases it usually helps to use a JSON navigator. I like this one: https://jsoneditoronline.org/.
  • When you filter responses, pay attention to small differences in your JMESPath syntax. The following two expressions will yield almost the same result, as both expressions produce a column of dates and a column of time zones. However they have an important difference:
    1. This first expression evaluates each dataset separately. This means that any missing or null values will be ignored, such that populated data shifts up into the first open cell, and the two data sets may not match up across columns:
      {date:data[].date,timezone:data[].timezone}
    2. This second expression will evaluate the entire data array, and keep these elements synced between columns. Therefore this is almost always the better option:
      data[].{date:date,timezone:timezone}

Troubleshooting

  • In some cases, you’ll find that an expression like properties[].{field1:field1,field2:field2} doesn’t always produce new fields for field1 and field2, resulting in mismatched columns. This can happen when the array (“properties” in this example) is not consistently populated, since the expression doesn’t work on a completely empty array. To resolve, modify the expression like this: properties[].{field1:field1,field2:field2} || {field1:"",field2:""}.
  • Occasionally you may run into JSON constructed of only objects. This is an undesirable data structure as it prevents API Connector from determining which data points should be returned as rows vs. columns (it typically results in all data on a single row, with a new column header for each element). In those cases, one option is to convert the data into a more useful structure with the values(@) function, e.g. Data.values(@) if Data is the name of the parent object.
  • In the case that your JSON keys start with a number or contain dashes, you’ll need to enclose them in quotes, e.g. "123"[].{order_id:order_id}. To avoid issues, you can enclose all keys with quotation marks, even those that don’t strictly require it.
  • If you enter an invalid JMESPath expression you may see error messages like “Request failed: Server response not in JSON, XML, or CSV format” or “Expected argument of type object, but instead had type X“. To address, make sure your JMESPath is correct and returns a valid JSON object (e.g. if you have a field named ID, add a JMESPath query of {id:id} rather than simply id).

JMESPath Info & Expression Tester

You may use any features of the JMESPath query language for JSON. Check the tutorial for additional methods of retrieving specific values from JSON: https://jmespath.org/tutorial.html

If you would like to test your JMESPath syntax, you can paste over the JSON examples in the tutorial and enter your own query into the input field, like this:

filter-fields-img5
Previous Connect to an OAuth2 API
Next ImportAPI Custom Function

57 thoughts on “Filter Fields (JMESPath)”

    • Yeahhh! Glad you discovered it πŸ˜€ This is actually one of my favorite features, it has a little bit of a learning curve but is so useful.

      Reply
    • The IMPORTAPI() custom function will execute with JMESPath enabled, just as it would if it were executed from the sidebar, and you’ll see your JMESpath-filtered results in your sheet. The one thing to look out for, though, is that Google allows custom functions to run for only 30 seconds, while sidebar requests can run for 6 minutes (documentation). This means that if you have a particularly large or slow request, you might run into problems if running it as a custom function. Besides that, though, there should be no difference.

      Reply
  1. Hello! Would this be a good place to filter down to specific endpoint values, and not just specific endpoints?

    i.e: Only bring in plays with “Sacks” as a type:

    1Play:
    Type: Sack

    2Play:
    Type: Touchdown

    Thanks

    Reply
  2. Can i solve following case with JMESPath?

    Input JSON:
    [{"Country":"US","Address":[{"Street":"1212, Jackson Ave","Block":"100","City":"LA"},{"Street":"90 Mary St","Block":"92","City":"San Jose"}]},{"Country":"England","Address":[{"Street":"10, 1st Aven","Block":"87","City":"London"}]}]

    Desired output:

    [{"Street":"1212, Jackson Ave","Block":"100","City":"LA","Country":"US"},{"Street":"90, Mary St","Block":"92","City":"San Jose","Country":"US"},{"Street":"10, 1st Aven","Block":"87","City":"London","Country":"England"}]

    Basically i wanted to inject outer attribute to each slice

    Reply
    • Hey Mahe, not sure if this will help, but I played around a bit on the JMESPath site and got it to work for the 1st value in the array with this JMESPath:
      [].{Country:Country,Street:Address[0].Street,Block:Address[0].Block,City:Address[0].City}
      For better help I suggest checking the docs or the JMESPath Gitter chat.

      Reply
  3. I have:

    {
      "rows": [
        {
          "keys": [
            "2021-01-01"
          ],
          "clicks": 165,
          "impressions": 2382,
          "ctr": 0.069269521410579349,
          "position": 5.8350125944584379
        },
    ... Repeat
    ]}

    But I simply can`t have something like: rows[*].{key:keys[0],click:clicks} on your tool, it works on JMESPath website tho.

    Reply
    • Hey Diego, what kind of error are you getting? I just tested it myself and it seemed fine, I got the same results through API Connector as I do on jmespath.org.

      Reply
  4. {"status":1,"stats":[{"partition":{"day":7,"month":3,"year":2020,"client":{"cid":"5fdcaac1cfda9f6a5a2a9410","title":"abc"},"partner":{"pid":28,"name":"test"},"product":{"id":313,"external_id":"1234"},"p3":"abc"},"supply":{"raw":"45","uniq":"45"},"actions":{"total":{"t_amt":0,"t_count":0},"hold":{"h_amt":0,"h_count":0}},"imp":0,"fallback":0}]}

    Could you please help with the Xpath for the above JSON, I tried but its not working as expected

    Output : day,month,year,cid,pid,id,p3,raw,unique,t_amt,h_amt,imp,fallback

    Reply
    • Try this:
      stats[].[{partition:partition.{day:day,month:month,year:year,client_cid:client.cid,partner_id:partner.pid,product_id:product.id,p3:p3}},{supply:supply},{actions:actions.{t_amt:total.t_amt,h_amt:hold.h_amt}},{imp:imp,fallback:fallback}]

      Reply
  5. I need to validate that the hour Is equal to 11:04 using jmespath in:departureDateTime=2021-03-20T11:04:00
    could you please advise on how i can do it?
    Thanks

    Reply
  6. Hi i was wondering if there was a way to get information from two arrays in the json? Ex
    {
    data: [...],
    included: [...],
    }

    I need information from both, but I’m unsure how to use both simultaneously. I currently have:
    data[].{id:id,type:type,attributes:attributes.line_score}
    AND
    included[?type=='new_player'].{name:attributes.name,POS:attributes.position,TEAM:attributes.team}

    Sorry if I’m unclear

    Reply
  7. Hey Ana,
    I’m using api connector to pull shopify orders.
    Currently trying to apply jmespath filtering on a grid style report.

    The filtering works, but the output is no longer in ‘grid’ format.

    An order that used to span multiple rows prior to the filtering, now only takes up a single row. There’s data missing.

    Is there something i’m doing wrong?

    Reply
    • When you apply JMESPath, it not only filters the data but changes the structure of the JSON, so it’s possible that the new shape no longer works as a grid. I can’t really say anything specific without knowing how you’ve filtered it, would you be able to share your JMESPath so I can check it?

      Reply
      • thanks for the reply Ana,
        i think i understand.
        but just in case there’s a chance to salvage this method. here is the filter i’m using:
        orders[].{name:name,id:id,line_items_name:line_items.name}

        with this filter, line_items.name returns blanks.

      • Hey there, line_items is an array, but you’ve included it with a dot like an object.
        Can you please try this instead?
        orders[].{name:name,id:id,line_items:line_items[].{name:name}}

  8. Hello,
    I’d like to sort the column K in the Google sheet (direct from import) when data are comming from API conector.
    Is there a meaning to specify the sort column or A->Z filter on a specfic data ?
    Thank you very much,
    Laurent

    Reply
  9. Hello!
    First of all congrats for this great tool! We are so so happy working with!

    I would love to ask about this:
    "nationality":["Argentinean","Spanish"]

    How Can I filter only for the first value?

    Thank you!

    Reply
  10. hi,
    very interesting application I am testing it!

    Q: it is possible to carry out a search like the following
    data = {"card":[{"id":1,"P1":2},{"id":1,"P1":1},{"id":1,"P1":4}]}
    find = [1,4,6]

    card[?P1 =='1' || P1 =='4' || P1 =='6'].{id:id,P1 : P1}
    I m currently looking for it this way, is there another way where I can place a list of the matches?
    for sample
    card[?P! == find].{id:id,P1 : P1}
    or
    card[?P! == ([1,2,6])].{id:id,P1 : P1}
    thanks for your help
    regards

    Reply
    • In addition to the first way you suggested, you can do it this way:
      card[?contains(`["1", "4","6"]`, P1)].{id:id,P1:P1}

      There may be other ways, too, but those are the methods I’m familiar with.

      Reply
      • Excellent, you have clarified the doubts … this response was quite quick and function correctly Thx…
        Regards.

  11. I need to connect with a Coinbase account, only getting the CONFIRMED/COMPLETED payment requests to a google sheet (expired ones are not useful). Does someone know how to write the path? Or either how to extend the limit beyond 100 lines (i have more than 100 payment requests)

    Reply
    • Can’t really say much without knowing the structure of the data, can you please share some sample JSON or the URL you’re calling?

      Reply
  12. The output structure is quite simple, just a grid where each line is a payment request and on the right there is a column which says “COMPLETED” or is in blanks. I need to have only the Completed ones.

    The standard API Response shows something like this:

    },
    {
    "time": "2017-01-31T20:50:02Z",
    "status": "COMPLETED"
    },
    {

    Data taken from: https://commerce.coinbase.com/docs/api/#list-charges

    Reply
    • Thanks for the link, based on their documentation you can use the following query:
      data[?timeline[?status=='COMPLETED']]

      Also, you can get more than 100 records by pagination like this:
      API URL: https://api.commerce.coinbase.com/charges?limit=100
      Pagination: next page URL
      Field name: next_uri
      Number of pages: 5
      That would get you 500 records. The JMESPath filter gets applied after so you’d see fewer records than that in your sheet if you’re using both together.

      Reply
      • The “COMPLETED” filter is working 100%, thank you so much!

        Regarding the amount of records, i’m still getting 100 records even with the parameters you’ve provided. The “Pagination Total” column tells me there are 126 total records so i’m still missing 26. I believe maybe Coinbase sets a 100 lines limit which cannot be changed?

      • Cool, I’m glad that worked. Their record limit can’t be changed, but API Connector’s pagination function will automatically cycle through and grab 100 records at a time.
        For field name, can you please try pagination.next_uri? (instead of next_uri). I didn’t notice the ‘pagination’ parent node before.

  13. Excellent! The filter and the records’ extension have now both worked. I spent days trying to make it work. So grateful. Thank you!

    Reply
      • Hey Ana, i’ve been reviewing the work done with the filters you provided and i realized i need to filter not only “COMPLETED” lines but also “RESOLVED” ones (this means, lines that are either COMPLETED or otherwise RESOLVED). Is there a way to add one more requirement to the “data[?timeline[?status==’COMPLETED’]]” you provided last week?

        Thanks!

  14. Hi ,

    Can you just get the raw value?

    "data": [
    {
    "id": 1,
    "name": "tester",
    "location": "Ottawa",
    "matesite": null
    ]
    }

    jmespath.search(“data[].location”, json)

    It returns [u’Ottawa’], but I just want to a raw value, which is Ottawa.

    I can achieve this with jq -r, but how to do with jmespath.search in python?

    Thanks!

    Reply
  15. Hi Ana,
    The preview is in XML. Does it change anything for the filtering ?
    How can I convert XML to JSON ?
    In my example below, what would be the structure
    to filter only on ID ?

    From a URL : /proposal_sheets.xml

    Reply
    • JMESPath is a filtering language for JSON only, so generally it would not work with XML. If possible you should look for a JSON endpoint.

      With that said, if they don’t provide a JSON endpoint, I have a potential workaround for you. It only works because when API Connector processes an XML response, it first converts it to JSON (so please don’t try using JMESPath on XML outside of API Connector as it probably wouldn’t work).
      What you can do is use dot notation to access any nested element. The only time you’ll need to use array syntax is if you see type="array" within your XML.

      Reply
  16. Hi
    I have a rest body similar to this example:

    {​​​​​
    "machines": [
    {​​​​​"name": "a", "type": "1","state": "running"}​​​​​,
    {​​​​​"name": "b","type": "2", "state": "stopped"}​​​​​,
    {​​​​​"name": "b","type": "1", "state": "running"}​​​​​
    ]
    }​​​​​

    how can i get the element from machines where name=a and state=running, in other words, filtering on multiple items of an element .

    Thanks a lot

    Reply
  17. Hi, I’m trying to filter out results by ‘type’ from a website API but I keep getting an error ‘Request failed: Server response not in JSON, XML, or CSV format’ on API connector. Not sure if the filtering fits correctly here
    type[?type==’fire’].config.stats.type

    My JMESPath:
    data[].{id:id,tokenId:tokenId,Price:startingPrice,Available:status,type[?type=='fire'].config.stats.type,LE:config.farm.le,Hours:config.farm.hours}

    from JSON :
    data
    {"id":1000713156,"tokenId":224271,"startingPrice":67,"status":1,"config":{"visual":{},"stats":{"type":"fire"},"farm":{"le":431,"hours":48}}}

    Thanks!

    Reply
    • This is the one of the entries in JSON i’d like to filter out of.
      The only difference here is ‘light’. But i’d like to filter out by specific type e.g fire,ice

      {"status":0,"data":[{"id":1011910930,"tokenId":534479,"ownerId":"0x9cf94145340f8e48ea591224466420f5f61086ba","plantTokenId":534479,"timeSell":1632271324,"startingPrice":47,"endingPrice":47,"duration":1718671316653,"status":1,"updatedAt":1632271324,"iconUrl":"https://pvuresources.s3.ap-southeast-2.amazonaws.com/icon/plant/19_1.png","config":{"visual":{},"stats":{"type":"light","hp":0,"defPhysics":0,"defMagic":0,"damagePhysics":0,"damageMagic":159,"damagePure":0,"damageHpLoss":0,"damageHpRemove":0},"farm":{"le":1209,"hours":240}}}]}

      Reply
    • Your filter doesn’t look quite right, can you please try this instead?
      data[?config.stats.type=='fire'].{id:id,tokenId:tokenId,Price:startingPrice,Available:status,type:config.stats.type,LE:config.farm.le,Hours:config.farm.hours}

      Reply
      • Yes! Works like a charm, thank you!

        How can I add aditional types? For instance ‘ice’?

        data[?config.stats.type==’fire’ && ‘ice’]. ?

      • Hi, again I figured it out how to add additional types with OR | |.

        data[?config.stats.type==’electro’||config.stats.type==’fire’||config.stats.type==’ice’].{,,,.,,,}

        Thanks again

Leave a Comment

Table of Contents