Home

Tricks for exploring Twarc json with jq

These are some snippets which are handy for working out the json structure of Twarc data. The snippets should work in bash or other similar shells, where jq (a very useful json manipulating utility) is installed (see the jq installation docs).

To get a list of all the tweet object keys in the first page of results:

head -n 1 tests/data/ObservatoryTeam.jsonl | jq '.data | map(keys) | flatten | unique'

To get a list of all keys within objects within arrays within objects within arrays...:

head -n 1 tests/data/ObservatoryTeam.jsonl | jq '[.includes.users[].entities | objects | keys] | flatten | unique'