Using cursors to navigate collections

Cursoring

The Twitter REST API utilizes a technique called ‘cursoring’ to paginate large result sets. Cursoring separates results into pages (the size of which are defined by the count request parameter) and provides a means to move backwards and forwards through these pages.

To retrieve cursored results, you initially pass a cursor with a value of -1 to the endpoint. By default, an API endpoint that supports cursoring will assume -1 was passed as cursor if you do not provide one. The response to a cursored request will contain previous_cursor, next_cursor, previous_cursor_str and next_cursor_str. As is the case with most ID values in Twitter’s APIs, the _str values are provided for languages that cannot support large integers (e.g. JavaScript).

The next_cursor is the cursor that you should send to the endpoint to receive the next batch of responses, and the previous_cursor is the cursor that you should send to receive the previous batch. You will know that you have requested the last available page of results when the API responds with a next_cursor = 0.

Pseudocode

The pseudocode to iterate over all responses from a cursored endpoint is:

cursor = -1
api_path = "https://api.twitter.com/1.1/endpoint.json?screen_name=targetUser"

do {
    url_with_cursor = api_path + "&cursor=" + cursor      
    response_dictionary = perform_http_get_request_for_url( url_with_cursor )
    cursor = response_dictionary[ 'next_cursor' ]
}
while ( cursor != 0 )

Example

Let’s see a real-life example of cursors in action. Consider the common scenario where we want to obtain the list of IDs who follow a user who has a large amount of followers. Instead of returning all IDs in one response set, the endpoint will instead return pages of results.

GET https://api.twitter.com/1.1/followers/ids.json?screen_name=theSeanCook

{
    "ids": [
        385752029, 
        602890434, 
        ...
        333181469, 
        333165023
    ], 
    "next_cursor": 1374004777531007833, 
    "next_cursor_str": "1374004777531007833", 
    "previous_cursor": 0, 
    "previous_cursor_str": "0"
}

We now have a means to move forwards through our dataset, via the next_cursor. To get the next batch of at least 5,000 results, perform the same request, but set the cursor to the value of the next_cursor.

GET https://api.twitter.com/1.1/followers/ids.json?screen_name=theSeanCook&cursor=1374004777531007833

{
    "ids": [
        333156387, 
        333155835, 
        ...
        101141469, 
        92896225
    ], 
    "next_cursor": 1323935095007282836, 
    "next_cursor_str": "1323935095007282836", 
    "previous_cursor": -1374003371900410561, 
    "previous_cursor_str": "-1374003371900410561"
}

Notice that we now have a next_cursor as well as a previous_cursor. This means that we can now move back and forth through our results. Let’s keep advancing through the data with the next cursor.

GET https://api.twitter.com/1.1/followers/ids.json?screen_name=theSeanCook&cursor=1323935095007282836

{
    "ids": [
        73635015, 
        61175149, 
        ...
        18300955, 
        32684766
    ], 
    "next_cursor": 0, 
    "next_cursor_str": "0", 
    "previous_cursor": -1323886329961827926, 
    "previous_cursor_str": "-1323886329961827926"
}

Huzzah. The next_cursor is now 0, which indicates that there are no more remaing pages. We’ve now completed iterating over the accounts’ followers.