OMG! Twitter release an OFFICIAL conversations API!

One of the most requested Twitter API features is now available - the ability to get replies to a Tweet as a thread.

Long time readers know that I've long been a fan of Visualising Twitter Conversations in 2D Space. But up until now you had to use horrible hacks to get the data. As trailed in their recent blogpost - conversation threading is now part of the official API!

This allows you to understand a Twitter conversation as a Tree:

1291422339075313664
├── 1291425742908207107
│   ├── 1291426352642621440
│   │   ├── 1291426673049718784
│   │   └── 1291427069474287616
│   └── 1291750186499239937
│       └── 1291751567947902977
│           └── 1291754176901341184
│               └── 1291755872826392576
│                   └── 1291756031454908417
│                       └── 1291756783480922112
├── 1291444493246398464
│   └── 1291450004956155905
├── 1291641587819253760
├── 1291648152844599297
│   └── 1291793510685868033
├── 1291722705167953921
└── 1291997111999848448

Quickstart

I'm going to assume you already have twurl installed and configured. If not, there's a quick guide at the end of this blog post.

twurl -j "/2/tweets/search/recent?query=conversation_id:1293593516040269825&tweet.fields=in_reply_to_user_id,author_id,created_at,conversation_id&max_results=100"

This takes a Tweet with ID 1293593516040269825 and returns all the replies to it - and replies to those replies! You also get some metadata about the conversation. It uses the new search API.

(Note: This example may not work for you. I think the search is limited to the last 7 days.)

Here's a short conversation

{
  "data": [
    {
      "author_id": "178648759",
      "created_at": "2020-06-10T14:50:44.000Z",
      "id": "1270730119460642816",
      "referenced_tweets": [
        {
          "type": "replied_to",
          "id": "1270692123516567553"
        }
      ],
      "text": "@iotwatch @WordPress I hear you. I'm in the process of ditching WordPress altogether. Making a new website with @GetPublii instead. Not as many features as wp obviously, but the new site is going to be so much faster!"
    },
    {
      "author_id": "156967608",
      "created_at": "2020-06-10T12:29:31.000Z",
      "id": "1270694583043870720",
      "referenced_tweets": [
        {
          "type": "replied_to",
          "id": "1270693741930647552"
        }
      ],
      "text": "@edent @WordPress I mean I am not a programmer but bust my chops learning how to install a .org instance on my own domain, tweak CSS and PHP because the payback was a simple and Word-like writing experience. Blocks is b(ul)locks."
    },
    {
      "author_id": "14054507",
      "created_at": "2020-06-10T12:26:11.000Z",
      "id": "1270693741930647552",
      "referenced_tweets": [
        {
          "type": "replied_to",
          "id": "1270692123516567553"
        }
      ],
      "text": "@iotwatch @WordPress completely agree. I've installed the classic editor and won't be going back. I've seen newbies try to use blocks and it always seems to end in tears."
    },
    {
      "author_id": "757222",
      "created_at": "2020-06-10T12:20:59.000Z",
      "id": "1270692436004700160",
      "referenced_tweets": [
        {
          "type": "replied_to",
          "id": "1270692123516567553"
        }
      ],
      "text": "@iotwatch @WordPress Not a fan?"
    }
  ],
  "includes": {
    "users": [
      {
        "id": "178648759",
        "name": "Bart.",
        "username": "filmbarend"
      },
      {
        "id": "156967608",
        "name": "Alexandra D-S",
        "username": "iotwatch"
      },
      {
        "id": "14054507",
        "name": "Terence Eden",
        "username": "edent"
      },
      {
        "id": "757222",
        "name": "Piers",
        "username": "PiersTincknell"
      }
    ]
  },
  "meta": {
    "newest_id": "1270730119460642816",
    "oldest_id": "1270692436004700160",
    "result_count": 4
  }
}

Limitations

You will need to be enrolled in the V2 API.

This is a beta service with limitations on API use. Additionally, I doubt it will work when you have a thousand people angrily responding to a Tweet.

The Tweets are unordered, so you'll need to create a datastructure if you want to do any visualisations like this:

This is limited to your app's API key. So, you're not going to create an amazing new way to interact with Twitter just yet - but you can start experimenting.

Oh, and you don't get the "head" of the conversation. So you'll need to request the originating tweet separately.

Set up

I'm assuming you're using Linux. First, install Ruby:

sudo apt install ruby

Then, install twurl:

sudo gem install twurl

If you haven't already, create a Twitter API key and apply for Twitter Labs access. You will need to enable Labs access to Recent Search.

Once you have an API key, authorise twurl:

twurl authorize --consumer-key ABC123 --consumer-secret XYZ789

Click on the link it generates, sign in, it will give you a PIN, return to the terminal and paste it in.

Finally, give twurl bearer authorisation.

twurl authorize --bearer --consumer-key ABC123 --consumer-secret XYZ789

Done! You can now make requests to the Twitter Labs using twurl. Aren't you lucky!

Python Trees

Here's a scrap of Python which uses TreeLib to create a tree structure for understanding the conversation.

I've assumed you've saved twurl's output as tweets.json.

import json
 
# Load the data
with open("tweets.json", 'r') as f:
    data = json.load(f)
 
#  Get just the tweets
tweets = data["data"]
 
#  Get ID - for sorting
def getID(t):
    return t.get("id")
 
#  Sort the tweets
tweets.sort(key=getID)
 
#  Tree Structure
from treelib import Node, Tree
tree = Tree()
 
#  First Parent is the root
rootID = tweets[0]["referenced_tweets"][0]["id"]
tree.create_node(rootID, identifier=rootID, data="root")
 
#  Load the Tweets into a Tree Structure
for tweet in tweets:
    tweetID = tweet["id"]
    parentID = tweet["referenced_tweets"][0]["id"]
    tree.create_node(tweetID, identifier=tweetID, data=tweet, parent=parentID)
 
#  Display the Tree
tree.show()