Authenticating with the Twitter API
The first step is to authorise access to the Twitter API. This
requires completion of an application and approval for Twitter
Developer access. Once the application has been approved then the
Twitter Developer Portal will become available and a new “Standalone
App” can be created. Creation of an app allows keys to be generated
through which software such as vosonSML can access and
collect data from the Twitter v1.1 API.
vosonSML is only concerned with reading Twitter data, so
a full range of typical third-party app user or
bot functions are not required. The ideal type of
authentication for read-only access is application based
authentication with a bearer token that can be generated
and found under the Authentication Tokens section of a
developers app project on the Developer Portal. This type of token has
higher tweet collection rate-limits than other methods but less
permissions. With a bearer token the Twitter authentication
object can be set as follows:
twitterAuth <- Authenticate("twitter", bearerToken = "xxxxxxxxxxxx")A developer access token can also be created by using
all of the developer keys generated for an app. This type of access is
suited to automated software such as bots that are required
to undertake a range of read and write Twitter activities. A
developer access token is not required for
vosonSML but can be used if desired, it will however have
lower rate-limits than the bearer token:
twitterAuth <- Authenticate(
"twitter",
appName = "My App",
apiKey = "xxxxxxxx",
apiSecret = "xxxxxxxx",
accessToken = "xxxxxxxx",
accessTokenSecret = "xxxxxxxx"
)There is also a user based access method available in
which a Twitter user can permit an app to access the API on their
behalf. This is the method third-party Twitter clients use to allow
users to perform Twitter functions such as posting tweets with their
software. When authorizing the software to their account the user will
be informed of the scope of permissions they are granting to the app.
This method has the advantage of per-user rather than application
rate-limits. This method is currently only available to be used with
vosonSML if the user has access to an app
API key and API secret:
twitterAuth <- Authenticate(
"twitter",
appName = "An App",
apiKey = "xxxxxxxxxxxx",
apiSecret = "xxxxxxxxxxxx"
)In all cases, Twitter authentication creates an authentication object with access token that can be re-used in the future by saving it to disk:
saveRDS(twitterAuth, file = "twitter_auth")The following loads into the current session a previously-created authentication object:
twitterAuth <- readRDS("twitter_auth")Collecting Data from Twitter
The syntax for collecting Twitter data follow Twitter’s rules and filtering documentation. It is possible to collect tweets including particular terms (e.g. hashtags), and boolean searches (see standard search operators) are also possible. The collection may be filtered by, for example, type of Twitter activity (e.g. to include retweets only), number of collected tweets, language of tweet. As an example, the following collects 1000 recent tweets containing the ‘#auspol’ hashtag (a prominent hashtag for Australian politics), with retweets not being collected.
twitterData <- twitterAuth |>
Collect(
searchTerm = "#auspol",
numTweets = 1000,
includeRetweets = FALSE,
writeToFile = TRUE,
verbose = TRUE
)The Collect() function takes the following arguments
(when used for collecting Twitter data): credential (object
generated from Authenticate() with class name
“twitter”(above we pass this via the pipe), searchTerm
(character string that specifies a Twitter search term),
searchType (character string indicating how to filter
returned tweets with options ‘recent’, ‘mixed’ or ‘popular’; default
type is ‘recent’), numTweets (numeric vector that specifies
how many tweets to be collected; default is 100),
includeRetweets (whether the search should filter out
retweets or not; default is TRUE), retryOnRateLimit
(whether to automatically pause collection when the Twitter API rate
limit is reached, and then restart; default is FALSE.),
writeToFile (whether to write the returned dataframe to
disk as an .rds file; default is FALSE), and
verbose (whether to output information about the data
collection; default is FALSE).
The Collect() function returns a tibble (an enhanced
dataframe which has features that can make working with the data
easier). We can view the data we just collected (the following has been
modified to anonymise the data):
> twitterData$tweets
# A tibble: 999 x 90
user_id status_id created_at screen_name text source
<chr> <chr> <dttm> <chr> <chr> <chr>
1 xxxxxx… xxxxxxxx… 2020-01-09 12:02:13 xxxx "htt… Twitt…
2 xxxxxx… xxxxxxxx… 2020-01-09 12:01:32 xxxxxxxxx "Fir… Twitt…
3 xxxxxx… xxxxxxxx… 2020-01-09 12:00:44 xxxxxxxxxxx "Ser… Twitt…
[snip]
… with 989 more rows, and 84 more variables: display_text_width <dbl>,
...Importing Twitter Data
If you are reading a previously saved writeToFile
Twitter dataframe from disk, you simply need to use the
readRDS function:
twitterData <- readRDS("2020-09-26_095354-TwitterData.rds")As vosonSML uses rtweet for data collection
you can also import rtweet tweet data from dataframe or
.RDS file using ImportRtweet():
# from dataframe
tweets <- rtweet::search_tweets("#auspol", n = 100)
twitterData <- tweets |> ImportRtweet()
# or from file
twitterData <- ImportRtweet("rtweet_search_n100.rds")Creating Twitter Networks
It is currently possible to create four types of networks using Twitter data: (1) actor network; (2) activity network; (3) 2-mode network and (4) semantic network.
Actor Network
In the Twitter actor network, nodes are users who have either tweeted using the target search terms (#auspol in the above example) or else are mentioned or replied to in tweets featuring the search terms. Edges represent interactions between Twitter users, and an edge attribute indicates whether the interaction is a mention, reply, retweet, quoted retweet or self-loop. Self-loop edges are created in two situations: (1) a user authors a tweet and mentions or replies to themself; (2) a user authors a tweet containing the search term, but does not mention or reply to any other user in that tweet.
actorNetwork <- twitterData |>
Create("actor", writeToFile = TRUE, verbose = TRUE)
actorGraph <- actorNetwork |> Graph(writeToFile = TRUE, verbose = TRUE)Create("actor") returns a named list containing two
dataframes named “nodes” and “edges” (the following has been modified to
preserve anonymity):
> actorNetwork
$edges
# A tibble: 1,725 x 5
from to edge_type timestamp status_id
<fct> <fct> <fct> <fct> <fct>
1 xxxxxxxx xxxxxxxx quote 2020-01-09 12:00… xxxxxxxxxxxx…
2 xxxxxxxx xxxxxxxxx quote 2020-01-09 09:37… xxxxxxxxxxxx…
[snip]
# … with 1,715 more rows
$nodes
# A tibble: 1,158 x 2
user_id screen_name
<fct> <fct>
1 xxxxxxxx xxxx
2 xxxxxxxx xxxxxxxxx
[snip]
# … with 1,148 more rows
attr(,"class")
[1] "list" "network" "actor" "twitter"This list is then passed to Graph(), which returns an
igraph graph object, and in the above example, the
writeToFile parameter is used to write the graph to file in
GraphML format. The following shows a summary of the
graph:
> actorGraph
IGRAPH bc177a6 DN-- 1158 1725 --
+ attr: type (g/c), name (v/c), screen_name (v/c), label (v/c),
| edge_type (e/c), timestamp (e/c), status_id (e/c)
+ edges from bc177a6 (vertex names):
[1] xxxxxxxx ->xxxxxxxx
[2] xxxxxxxx ->xxxxxxxxx
[snip]
+ ... omitted several edgesThe Twitter actor network contains a graph attribute
type which is set to “twitter” (this attribute is required
for VOSON Dashboard). The following node attributes are
collected from the Twitter profile data: name (Twitter ID),
screen_name (Twitter handle or screen name) and
label (a concatenation of the ID and screen name). The edge
attributes are: edge_type (whether the edge is a mention,
reply, retweet, quoted retweet or self-loop), timestamp
(when the tweet that led to the creation of the edge was authored) and
status_id (the Twitter ID for the tweet).
The example actor network contains 1158 nodes and 1725 edges. The
following code uses igraph functions to: (1) remove all
edges other than reply edges; (2) construct a subnetwork consisting of
the giant component (the largest set of connected nodes); (3) plot this
network.
library(igraph)
# remove edges that are not reply edges
g2 <- delete.edges(actorGraph, which(E(actorGraph)$edge_type != "reply"))
# get the giant component
cc <- clusters(g2)
g2 <- induced_subgraph(g2, which(cc$membership == which.max(cc$csize)))
# open and write plot to a png file
png("twitter_actor_reply_gc.png", width = 600, height = 600)
plot(g2, vertex.label = "", vertex.size = 4, edge.arrow.size = 0.5)
dev.off()
Twitter actor network - replies only - giant component
It is often useful to have the tweet text content in the network.
This can be achieved by using the status_id edge attribute
to pull the tweet text content from the dataframe returned by
Collect(), and store it as an edge attribute.
vosonSML makes this easy with the AddText()
function, with the following example creating a graph with an edge
attribute vosonTxt_tweet which stores the tweet text
content.
actorGraphWithText <- twitterData |>
Create("actor") |> AddText(twitterData) |> Graph()Now we have tweet text content stored as an edge attribute, we can
use it for text analysis or creating new node attributes. As an example,
the following creates a new node attribute tweetedBushfires
which has the value “yes” if the Twitter user authored at least one
tweet containing the word “bushfire” and “no” otherwise.
# get the index of nodes or users who tweeted the word "bushfire"
ind <- tail_of(
actorGraphWithText,
grep("bushfire", tolower(E(actorGraphWithText)$vosonTxt_tweet))
)
# set node attribute
V(actorGraphWithText)$tweetedBushfires <- "no"
V(actorGraphWithText)$tweetedBushfires[ind] <- "yes"The following creates and plots the giant component in the reply network, with red nodes indicating those users who tweeted using the term “bushfire”.
# remove edges that are not reply edges
g3 <- delete.edges(
actorGraphWithText, which(E(actorGraphWithText)$edge_type != "reply")
)
# get the giant component
cc <- clusters(g3)
g3 <- induced_subgraph(g3, which(cc$membership == which.max(cc$csize)))
# set node colour based on tweeted bushfires attribute value
V(g3)$color <- ifelse(V(g3)$tweetedBushfires == "yes", "red", "grey")
# open and write plot to a png file
png("twitter_actor_reply_gc_bushfires.png", width = 600, height = 600)
plot(g3, vertex.label = "", vertex.size = 4, edge.arrow.size = 0.5)
dev.off()
Twitter actor network - replies only - giant component (red nodes indicate who tweeted at least once about bushfires)
The igraph graph object can then be saved to disk as a
GraphML file using the igraph function
write.graph, and later imported into network analysis
software such as igraph, VOSON Dashboard and
Gephi:
# save the graph as a graphml file
write.graph(g3, "twitter_reply_gc_bushfires.graphml", format = "graphml")Finally, the AddUserData() function can be used to
create additional node attributes from the Twitter profile, for example,
number of followers and number of tweets authored by the user.
Note that by default, vosonSML will only collect Twitter
profile data for those users who authored tweets that have been
collected. That is, in the above example, profile data will not be
collected (by default) for a user who was mentioned in a tweet that
contained #auspol, but did not author such a tweet. The
lookupUsers argument can be used to make additional Twitter
API calls and collect the profile data for mentioned users whose profile
data is absent, so that their additional profile attributes can also be
added:
# create an actor network with user metadata
actorGraphWithUserAttr <- actorNetwork |>
AddUserData(twitterData, lookupUsers = TRUE, twitterAuth = twitterAuth) |>
Graph(writeToFile = TRUE)Activity Network
In the Twitter activity network, nodes represent tweets and edge types are: replies, retweets and quoted retweets.
# create an activity network with tweet text
activityNetwork <- twitterData |> Create("activity") |> AddText(twitterData)
activityGraph <- activityNetwork |> Graph(writeToFile = TRUE)Create("activity") returns a named list containing two
dataframes named “nodes” and “edges” (the following has been modified to
preserve anonymity):
> activityNetwork
$nodes
# A tibble: 1,408 x 5
status_id user_id screen_name created_at vosonTxt_tweet
<chr> <chr> <chr> <chr> <chr>
1 xxxxxxxxxxx… xxxxxxxx xxxx 2020-01-09 … "xxxxxxxxxxxxxxxxxxxxx…
2 xxxxxxxxxxx… xxxxxxxx xxxxxxxxx 2020-01-09 … "xxxxxxxxxxxxxxxxxxxxx…
[snip]
# … with 1,398 more rows
$edges
# A tibble: 662 x 3
from to edge_type
<chr> <chr> <chr>
1 xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx quote
2 xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx quote
[snip]
# … with 652 more rows
attr(,"class")
[1] "list" "network" "activity" "twitter" "voson_text"Note that in the above, AddText() was used to add the
comment text data to the network dataframe, stored as a node attribute.
This list is then passed to Graph(), which returns an
igraph graph object (the following has been
anonymised):
> activityGraph
IGRAPH e60c486 DN-- 1408 662 --
+ attr: type (g/c), name (v/c), user_id (v/c), screen_name (v/c),
| created_at (v/c), vosonTxt_tweet (v/c), label (v/c), edge_type (e/c)
+ edges from e60c486 (vertex names):
[1] xxxx->xxxx
[2] xxxx->xxxx
[snip]
+ ... omitted several edgesThe Twitter activity network contains a graph attribute
type (set to “twitter”). The node attributes are:
name (Twitter ID for the tweet), user_id
(Twitter ID for the user who authored the tweet),
screen_name (Twitter handle or screen name of the user who
authored the tweet), created_at (timestamp when the tweet
was authored), vosonTxt_tweet (text content of the tweet)
and label (a concatenation of name and
user_id). The edge attribute is edge_type
which can have the value ‘reply’, ‘retweet’ or ‘quote’.
The example Twitter activity network contains 1408 nodes and 662 edges. The following is a visualization of the network, where nodes are tweets and tweets mentioning bushfires are indicated in red.
# create a subgraph containing nodes of components that have more than 5 nodes
cc <- clusters(activityGraph)
g4 <- induced_subgraph(
activityGraph,
which(cc$membership %in% which(cc$csize > 5))
)
# set node colour based on if tweet contains the word "bushfire"
ind <- grep("bushfire", tolower(V(g4)$vosonTxt_tweet))
V(g4)$color <- "grey"
V(g4)$color[ind] <- "red"
# open and write plot to a png file
png("twitter_activity.png", width = 600, height = 600)
plot(g4, vertex.label = "", vertex.size = 4, edge.arrow.size = 0.5)
dev.off()
Twitter activity network (red indicates tweets mentioning bushfires)
It should be noted that a limitation of the Twitter API is that retweet chains are not provided. This means that if user i tweeted an original tweet, and then user j retweeted this tweet, and user k retweeted j’s retweet, the activity network will show edges connecting the two retweets to the original tweet.
2-mode Network
In the Twitter 2-mode network, the two types of nodes are actors (Twitter users) and hashtags. There is an edge from user i to hashtag j if user i authored a tweet containing hashtag j.
# requires the tidytext package for tokenizing text
install.packages("tidytext")
# create a 2-mode network with the hashtag "#auspol" removed
twomodeNetwork <- twitterData |>
Create("twomode", removeTermsOrHashtags = c("#auspol"))
twomodeGraph <- twomodeNetwork |> Graph()Create("twomode") returns a named list containing two
dataframes named “nodes” and “edges” (the following has been modified to
preserve anonymity). Note that in this example, the
removeTermsOrHashtags argument was used to exclude
‘#auspol’, since by construction all tweets contained this hashtag.
> twomodeNetwork
$nodes
# A tibble: 1,146 x 2
entity_id display_name
<chr> <chr>
1 xxxxxxxx xxxx
2 xxxxxxxx xxxxxxxxx
3 #auspol2020 #auspol2020
4 #australianbushfiredisaster #australianbushfiredisaster
[snip]
# … with 1,136 more rows
$edges
# A tibble: 1,675 x 5
from to edge_type timestamp status_id
<fct> <fct> <fct> <fct> <fct>
1 xxxxxxxx #auspol2020 hashtag 2020-01-09 12:0… xxxxxxxxxxxx…
2 xxxxxxxx #australianbushfiredis… hashtag 2020-01-09 12:0… xxxxxxxxxxxx…
[snip]
# … with 1,665 more rows
attr(,"class")
[1] "list" "network" "twomode" "twitter"This list is then passed to Graph(), which returns an
igraph graph object (this has been anonymised):
> twomodeGraph
IGRAPH 68bd240 DN-- 1146 1675 --
+ attr: type (g/c), name (v/c), display_name (v/c), label (v/c),
| edge_type (e/c), timestamp (e/c), status_id (e/c)
+ edges from 68bd240 (vertex names):
[1] xxxx -> #auspol2020
[2] xxxx -> #australianbushfiredisaster
[snip]
+ ... omitted several edgesThe Twitter 2-model network has a graph attribute type
(set to “twitter”). The node attributes are: name (hashtag
or Twitter user ID), display_name (hashtag or Twitter
handle or screen name), label (for users, a concatenation
of name and display_name, while for hashtags
it is name). The edge attributes are:
edge_type (‘hashtag’), timestamp (timestamp of
the tweet that led to the edge), status_id (Twitter ID of
the tweet that led to the edge).
# get index of nodes that are in the top 5 by highest in-degree
# this is the top 5 used hashtags, as all users have 0 in-degree
# in this network
ind <- order(degree(twomodeGraph, mode = "in"), decreasing = TRUE)[1:5]
# get index of nodes with an edge directed to the top 5 hashtags
# this is users who have tweeted with these hashtags
ind2 <- unlist(
lapply(ind, function(x) neighbors(twomodeGraph, x, mode = "in"))
)
# create a subgraph containing only the top 5 used hashtags and related users
g5 <- induced_subgraph(twomodeGraph, c(ind, as.numeric(ind2)))
# set node colour and label based on in-degree
# only hashtag nodes are set to blue and with label attribute set
V(g5)$color <- "grey"
V(g5)$color[which(degree(g5, mode = "in") > 0)] <- "blue"
V(g5)$label2 <- ifelse(degree(g5, mode = "in") > 0, V(g5)$label, "")
# open and write plot to a png file
png("twitter_twomode.png", width = 600, height = 600)
plot(g5, vertex.label = V(g5)$label2, vertex.size = 4, edge.arrow.size = 0.5,
vertex.label.cex = 1.8, vertex.label.color = "red")
dev.off()
Twitter 2-mode network - top-5 hashtags, and the users who tweeted them
Semantic Network
In the Twitter semantic network, nodes represent entities extracted from the tweet text: common words, hashtags and usernames. Edges reflect co-occurrence i.e. there is an edge between entities i and j if they both occurred in the same tweet.
# additional required packages for tokenization and stopwords
install.packages(c("tidytext", "stopwords"))
# create a semantic network with some common terms removed
# include only the top 5% occurring terms in the network
semanticNetwork <- twitterData |> Create(
"semantic",
removeTermsOrHashtags = c("#auspol", "auspol", "australia"),
termFreq = 5
)
# create an undirected graph
semanticGraph <- semanticNetwork |> Graph(directed = FALSE)Create("semantic") returns a named list containing two
dataframes named “nodes” and “edges”:
> semanticNetwork
$nodes
# A tibble: 799 x 1
value
<fct>
1 just
2 one
3 fire
4 going
5 still
6 hard
7 trying
8 since
9 try
10 sick
# … with 789 more rows
$edges
# A tibble: 10,990 x 3
from to weight
<fct> <fct> <int>
1 #auspol2020 #australianbushfiredisaster 2
2 #auspol2020 government 2
3 #auspol2020 fire 4
4 #auspol2020 australian 2
5 #auspol2020 bushfire 2
6 #auspol2020 fires 4
7 #auspol2020 #australiafires 1
8 #auspol2020 #australianbushfiresdisaster 1
9 #auspol2020 #australia 4
10 #auspol2020 bushfires 2
# … with 10,980 more rows
attr(,"class")
[1] "list" "network" "semantic" "twitter"The removeTermsOrHashtags argument is used to exclude
terms that we can expect to occur frequently (e.g. because of the
parameters used for the data collection). The termFreq
parameter is used to limit the network to the top 5 percent most
frequently occurring entities.
This list is then passed to Graph(), which returns an
igraph graph object:
> semanticGraph
IGRAPH cb8c381 UNW- 799 10990 --
+ attr: type (g/c), name (v/c), label (v/c), weight (e/n)
+ edges from cb8c381 (vertex names):
[1] #australianbushfiredisaster --#auspol2020
[2] government --#auspol2020
[3] fire --#auspol2020
[4] australian --#auspol2020
[5] bushfire --#auspol2020
[6] fires --#auspol2020
[7] #australiafires --#auspol2020
[8] #australianbushfiresdisaster--#auspol2020
+ ... omitted several edgesThe Twitter semantic network node contains a graph attribute
type (set to “twitter”). The node attributes are:
name (the entity i.e. hashtag or word), label
(same as name). The edge attribute is weight
(how many tweets the two entities co-occurred in).
The example Twitter semantic network has 799 nodes and 10990 edges. The following produces a visualisation of subnetwork of terms that contain the string “bushfire”:
# get index of the nodes whose term contains "bushfire"
ind <- grep("bushfire", tolower(V(semanticGraph)$name))
# create a subgraph containing only bushfire terms
g6 <- induced_subgraph(semanticGraph, ind)
# open and write plot to a png file
# plotted with large-graph-layout algorithm and edge weights
png("twitter_semantic.png", width = 600, height = 600)
plot(g6, layout = layout_with_lgl(g6), vertex.shape = "none", vertex.size = 4,
edge.width = 1 + log(E(g2)$weight))
dev.off()
Twitter semantic network - entities containing the word “bushfire” (edge thickness proportional to edge weight)
