epitweetr: user documentation

Epidemic Intelligence at ECDC

Article 3 of the European Centre for Disease Prevention and Control (ECDC) funding regulation and the Decision No 1082/2013/EU on serious cross-border threats to health have established the detection of public health threats as a core activity of ECDC.

ECDC performs Epidemic Intelligence (El) activities aiming at rapidly detecting and assessing public health threats, focusing on infectious diseases, to ensure EU’s health security. ECDC uses social media as part of its sources to early detect signals of public health threats. Until 2020, the monitoring of social media was mainly performed through the screening and analysis of posts from pre-selected experts or organisations, mainly in Twitter and Facebook.

More information and an online tutorial are available:

EI sources

EI tutorial

Monitoring social media trends

Some signals are not detected at all or are not detected early enough by the methods explained above. The automated monitoring of social media (e.g. analysis of social media trends) allows the detection of signals that may not be detected by the monitoring of pre-selected accounts in social media, and it improves the timeliness of signal detection.

The analysis of trends from social media by topic, time and place generates relevant signals for early detection.

In 2019, ECDC developed a prototype of a manual R-based tool for the early detection of public health threats from Twitter data. The epitweetr package is an extension of this prototype, to allow a wider geolocation of tweets and greater automatisation, which first version was published in October 2020.

Objectives of epitweetr

The primary objective of epitweetr is to use the Twitter Standard Search API version 1.1 and Twitter Recent Search API version 2 in order to detect early signals of potential threats by topic and by geographical unit.

Its secondary objective is to enable the user through an interactive web interface to explore the trend of tweets by time, geographical location and topic, including information on top words and numbers of tweets from trusted users, using charts and tables.

Repository of epitweetr material and training

More information on epitweetr is available in the epitweetr GitHub discussions. This post contains a summary of links and materials of relevance for new users.

Prerequisites for running epitweetr

Before using epitweetr, the following items need to be installed:

• R version 3.6.3 or higher

• Java 1.8 eg. openjdk version “1.8” https://www.java.com/download/. The 64-bit rather than the 32-bit version is preferred, due to memory limitations. In Mac, also the Java Development Kit https://docs.oracle.com/javase/9/install/installation-jdk-and-jre-macos.htm]

• If you are running it in Windows, you will also need Microsoft Visual C++ which in most cases it is likely to be pre-installed:

  • Microsoft Visual C++ 2010 Redistributable Package (x64) [https://www.microsoft.com/en-us/download/confirmation.aspx?id=26999]

Prerequisites for some of the functionalities in epitweetr

• Pandoc, for exporting PDFs and Markdown

  • https://pandoc.org/installing.html

• Tex installation (TinyTeX or MiKTeX) (or other TeX installation) for exporting PDFs

  • Easiest: https://yihui.org/tinytex/ install from R, logoff/logon required after installation

  • https://miktex.org/download full installation required, logoff/logon required after installation

• Machine learning optimisation (only for advanced users)

  • Open Blas (BLAS optimizer), which will speed up some of the geolocation processes: https://www.openblas.net/ Installation instructions: https://github.com/fommil/netlib-Java

  • or Intel MKL

• A scheduler

  • If using Windows, you need to install the R package: taskscheduleR

  • If using Linux, you need to plan the tasks manually

  • If using a Mac, you need to plan the tasls manually

Extra prerequisites for R developers

If you would like to develop epitweetr further, then the following development tools are needed:

• Git (source code control) https://git-scm.com/downloads

• Sbt (compiling scala code) https://www.scala-sbt.org/download.html

• If you are using Windows, then you will additionally need Rtools: https://cran.r-project.org/bin/windows/Rtools/

External dependencies

epitweetr will need to download some dependencies in order to work. The tool will do this automatically the first time the alert detection process is launched. The Shiny app configuration page will allow you to change the target URLs of these dependencies, which are the following:

• CRAN JARs: Transitive dependencies for running Spark, Lucene and embedded scala code. [https://repo1.maven.org/maven2]

• Winutils.exe (Windows only) This is a Hadoop binary necessary for running SPARK locally on Windows [https://github.com/steveloughran/winutils/raw/master/hadoop-3.0.0/bin/winutils.exe].

Please note that during the dependencies download, you will be prompted: first to stop the embedded database and then to enable it again. If you are on Windows and you have activated the tasks using the ‘activate’ buttons on the configuration page, you can performs this tasks by disabling and enabling the tasks on the ‘Windows Task Scheduler’. For more information see the section ‘Setting up tweet collection and the alert detection loop’

Installing epitweetr from CRAN

After installing all required dependencies listed in the section “Prerequisites for running epitweetr”, you can install epitweetr:

install.packages(epitweetr)

Sys.getenv("JAVA_HOME")

library(epitweetr)
epitweetr_app("data_dir")

Migrating to epitweetr v2

Migrating epitweetr from previous versions (before January 2022) to version 2.0.0 or higher is possible without any data loss. In this section, we will describe the necessary steps to perform the migration.

This migration is not necessary if you are installing epitweetr for the first time.

In epitweetr v2, we redesigned the way how tweets and series are stored. In previous versions, tweets were saved as compressed JSON files and series as RDS data frames in ‘tweets’ and ‘series’ folder, respectively. In epitweetr v2 or higher, we have moved to a different storage system allowing epitweetr to work as a search engine and allowing efficient updates, deletions and faster aggregation of data. For doing so, data is stored using Apache Lucene indexes in the ‘fs’ folder. Note that during migration, Twitter data are moved to the ‘fs’ folder and series are left as it is. Epitweetr reports will combine data from older and new storage system.

If you have an existing installation that contains data in the previous format, you have to migrate it following the steps detailed in this section. This applies to any epitweetr version before v2.0.0. You can also check this by looking in ‘tweets/geo’ or ‘tweets/search’ folders. If there is a json.gz file, migration is needed.

The migration steps are the following:

• Make sure the ‘Data collection & processing’ and ‘Requirements & alerts’ pipelines are stopped (you can see the status in the Shiny configuration page). If you are using Windows and you have launched the tasks using the activate buttons, you have to manually open the Windows task scheduler and disable these two epitweetr tasks or force these to stop by using the ‘stop’ button in the configuratio page.
• Close the Shiny App.
• Install the new epitweetr version (if you have not yet done it).
• Go to the configuration page in the Shiny app.
• Click on the ‘Run dependencies’ button.
• Activate the ‘Requirements & alerts’ pipeline If you are in Windows, you can click on the activate button or you can run on a new R session executing the following command:

  • epitweetr::detect_loop("data_dir")

• In the line of the dependencies tasks under ‘Requirements & alerts’ table, there will be a message reporting the status of the migration and download of necessary dependencies from the selected maven mirror. It will indicate whether some of the epitweetr tasks must be stopped or activated to complete migration and download of dependencies.
• First, it will download the dependencies for which the embedded epitweetr database should be stopped. If the embedded database is running, a message will appear asking you to stop the database (‘epitweetr database’ pipeline).
• After all dependencies have been downloaded, a message will appear asking to activate the embedded database (‘epitweetr database’). Please refer to the section ‘Setting up tweet collection and the alert detection loop’ for details.
• Once the epitweetr database is running, the migration will start. Epitweetr will report the estimated remaining time to finish the migration. Be aware that depending on the size of your data folder, this can take several days for which the machine running epitweetr cannot be turned off.
• As files are migrated to the new storage system, these are moved from the folder ‘tweets/search’ and ‘tweets/geo’ to ‘tweets/search_archive’ and ‘tweets/geo_archive’.
• Once the migration is done, you can backup and delete the _archive folders.
• You can test that your data has been successfully migrated by searching for tweets in particular periods using the ‘Data protection’ page.
• Click on the button ‘Run GeoNames’ and ‘Run languages’ and wait for them to finish.
• You can now activate the ‘Data collection & processing’.

Setting up tweet collection and the alert detection loop

In order to use epitweetr, you will need to collect and process tweets, run the ‘epitweetr database’ and ‘Requirements & alerts’ pipelines. Further details are also available in subsequent sections of this user documentation. A summary of the steps needed is as follows:

• Launch the Shiny app (from the R console)

library(epitweetr)
epitweetr_app("data_dir")

• In the configuration page of the Shiny app, in the manual tasks of the “Requirements & alerts pipeline”, click on “Run dependencies”, “Run GeoNames” and “Run languages” (their status will change to “pending”). This allows the ‘Requirements & alerts’ pipeline to download the elements needed. As long as no languages are added and no updates are available in geonames.org, these tasks have to be run only the first time you install epitweetr.

• Set up the Twitter authentication using a Twitter account or a Twitter developer app, see section Collection of tweets>Twitter authentication for more details

• If you want to use the Twitter API v2, you have to request access from the Twitter developer portal. More information on [https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api]

• Activate the embedded database

  • Windows: Click on the “Epitweetr database” activate button

  • Other platforms: In a new R session, run the following command

library(epitweetr)
fs_loop("data_dir")

• You can confirm that the embeded database is running if the ‘epitweetr database’ status is “Running” in the Shiny app configuration page and “true” in the Shiny app troubleshoot page.

• Activate the tweet collection and data processing

  • Windows: Click on the “Data collection & processing” activate button

  • Other platforms: In a new R session, run the following command

library(epitweetr)
search_loop("data_dir")

• You can confirm that the tweet collection is running, if the ‘Data collection & processing’ status is “Running” in the Shiny app configuration page (green text in screenshot above) and “true” in the Shiny app troubleshoot page.

• Activate the epitweetr database

  • Windows: Click on the “epitweetr database” activate button

  • Other platforms: In a new R session, run the following command

library(epitweetr)
fs_loop("data_dir")

• You can confirm that the epitweetr database is active if the ‘epitweetr database’ status is “Running” in the Shiny app configuration page (green text in screenshot above) and “true” in the Shiny app troubleshoot page.

• Activate the ‘Requirements & alerts’ pipeline:

  • Windows: Click on the “Requirements & alerts pipeline” activation button
  • Other platforms: In a new R session run the following command

library(epitweetr)
detect_loop("data_dir")

• You can confirm that the ‘Requirements & alerts’ pipeline is running, if the ‘Requirements & alerts’ pipeline status is “Running” in the Shiny app configuration page and “true” in the Shiny app troubleshoot page.

• You will be able to visualise tweets after ‘Data collection & processing’ and ‘epitweetr database’ are activated and the languages task has finished successfully.

• You can start working with the generated signals. Happy signal detection!

For more details, you can go through the section How does it work? General architecture behind epitweetr, which describes the underlying processes behind the tweet collection and the signal detection. Also, the section “The interactive Shiny application (Shiny app)>The configuration page” describes the different settings available in the configuration page.

Collection of tweets

Geolocation

In a parallel process to the collection of tweets, epitweetr attempts to geolocate all collected tweets using a supervised machine learning process. This process runs automatically after tweets are collected.

epitweetr stores two types of geolocation for a tweet: tweet location, which is geolocation information within the text of a tweet (or a retweeted or quoted tweet), and user location from the available metadata. For signal detection, the preferred location is used (i.e., tweet location) while in the dashboard both types can be visualised.

Most frequent elements found in and extracted from tweets

Epitweetr counts five types of elements in tweets

• Words: Words present in tweets. Words are then aggregated in the topwords time series

• Hashtags: hashtags present in tweets. Hashtags are then aggregated in the hashtags time series

• URLs: URLs present in tweets. URLs are then aggregated in the URLs time series

• Entities: Entities are automatically extracted and identified by the Twitter API v2. These are then aggregated in the entities time series. This information is only available when the API v2 is activated and properly configured. As per Twitter documentation “Entities are comprised of people, places, products, and organizations […]. They are programmatically assigned based on what is explicitly mentioned in the tweet text”.

• Contexts: Contexts are automatically extracted and identified by the Twitter API v2. These are then aggregated in the contexts time series. This information is only available when the API v2 is activated and properly configured. As per Twitter documentation “[Context are] derived from the analysis of a tweet’s text and will include a domain and entity pairing which can be used to discover tweets on topics that may have been previously difficult to surface […]. [It uses] a list of 50+ domains to categorize tweets”.

Note that, differently to the other visualisation figures, the most frequent elements found in and extracted from tweets are always based on the geolocation related to “tweet location” and not to “user location” regardless of the filter selected in the dashboard. Topwords & hashtags and entities & contexts are shown in the same figure, respectively.

Signal detection

The main objective of epitweetr is to detect signals in the observed data streams, i.e. counts in the aggregated time series that exceed what is expected. For detecting signals, epitweetr uses an extended version of the EARS (Early Aberration Reporting System) algorithm (Fricker, Hegler, and Dunfee 2008), which in what follows is denoted by eears (extended EARS). This algorithm is part of the R package surveillance (Salmon, Schumacher, and Höhle 2016).

As a default it uses a moving window of the past seven days to calculate a threshold. If the count for the current day exceeds this threshold, then a signal is generated.

Folder structure

epitweetr stores and aggregates tweets. When launching the application, you have to designate the and the configuration in the “data folder” you have to designate when launching the application.

Within the data folder there are three JSON files:

• properties.json, generated from the information from the General properties of the Shiny app

• topics.json managed by the search loop: it keeps a track of tweet collection plans and progress

• tasks.json managed by the detect loop: it keeps information and status of the different tasks done by this process.

There are also the following subfolders:

• “fs”, which contains the Lucene indexes for storing tweets and aggregated time series

• “geo”, which contains the GeoNames data as text, index files, tweets waiting to be geolocated and settings about the geolocation algorithm.

• “alert-ml”, which contains the machine learning models used for alert classification

• “hadoop”, which contains Spark dependencies for Windows operating systems

• “jars”, which contains collections of Java dependencies needed in the geolocation and aggregate processes

• “languages”, which contains fasttext files indexes and models used to perform geolocation in the tweet text

• “stats”, which contains JSON files with statistics used to optimise the aggregate process by linking tweet files and posted dates of tweets

• “alerts”, which contains JSON files of alerts detected by the requirements & alerts pipeline

• “collections”, which contains the aggregations produced by epitweetr

Dashboard: The interactive user interface for visualisation

The dashboard is where you can interactively explore visualisations of tweets. It includes a line graph (trend line) with alerts, a map and top words, URLs, hashtags, entities and contexts of tweets for a given topic. After selecting the parameters, you have to click on the ‘Run’ button in order to see or refresh the report.

In order to interactively explore the data, you can select from several filters, such as topics, countries and regions, time period, time unit, signal confidence and days in baseline. After selecting the filters, you must click on ‘Run’ to see the outputs in the dashboard.

Note that whatever options/settings you select on the dashboard, will have no effect on the alert detection. The alert detection settings are all selected in the configuration page of the Shiny app.

The alerts page

The alerts page summarises the signals detected within the specified study period and provides functionality for adding annotations for training machine learning algorithms to classify alerts on user’s defined topics. This page is splitted on two sections

The geotag page

This page supports the user in improving the geolocation algorithm. Epitweetr will allow the users to download the data that was used to build the classifiers for location identification. The predefined annotations are based on location vs non location words. Locations are extracted from geonames.org data base. Non location words are obtained from common words on the downloaded models that are not present on GoNames. You will be able to add tweets to the annotation datanbase and to manually annotate them until you reach the expected level of performance.

To help you on the evaluation process epitweetr will calculate standard machine learning metrics for evaluating its capacity to identify the right location in texts. These metrics are separately calculated by different type of texts: Tweet location, Tweet text, Tweet user description and total.

The following controls are available:

• Tweets to add: Choose the number of tweets to add to the geolocation spreadsheet
• Geolocate annotations: Pressing this button will add the selected number of annotations to the epitweetr geolocation spreadsheet and evaluate the geolocation algorithm on the current annotations
• Download annotations: Download the current version of annotations
• Upload & evaluate annotations: Upload a new version of the geolocation annotations to improve annotations performance. The algorithms for location identification will be trained using the provided data. Also performance metrics will be calculated using 75% of the data for training and 25% for test. The final model is trained using all data.

The geolocation spreadsheet has the following columns:

• Type: This column is a reference for the user to understand the kind of annotation. It can be any of the following.
  • Text: A text that can contains locations inside it can be a tweet or a user description.
  • Location: A text containing a location.
  • Person: A known person e.g. presidents that can be used by epitweetr to detect countries. This data is not used for training, just for over-writing the algorithm behavior.
  • Demonym: A known word to associate to a country e.g. Chilean. This data is not used for training, just for over-writing the algorithm behavior.
• Text: The text associated to this annotation.
• Location in text: The subpart of the text that contains a location. If many locations are present, the same line can be duplicated with different values on this column. If this column is empty epitweetr assumes that the annotation is associated with all the text.
• Location yes/no: whether the current text contains a location (demonym and people are not considered as location).
• Associate country code: Whether epitweetr should memorize the annotated text and bypass the machine learning algorithm to always associate the given text to the country (or geonames city) indicated by the provided code.
• Associate with: Whether epitweetr should memorize the annotated text and bypass the machine learning algorithm to always associate the given text to the country (or geonames city) indicated by the provided name.
• Source: The source of the annotation, it can be any of: Epitweetr database (provided by epitweetr team), Epitweetr model (obtained automatically by epitweetr) or Tweet (obtained from a tweet).
• Tweet Id: The tweet id if the text is associated to a tweet.
• Lang: The language associated to the annotation.
• Tweet part: The tweet part if associated to a tweet. It can be any of: “text”, “user description” or “user location”.
• Epitweetr match: The location found by epitweetr
• Epitweetr country match: The country of the location found by epitweetr.
• Epitweetr country code match: The country code of the location found by epitweetr.

The data protection page

In the data protection page, the user can search, anonymise and delete tweets from the epitweetr database to support data deletion requests.

The following search filters are available: Topic, period, country & regions, mentioning (for selecting only mentions of the provided users), from users (for selecting only tweets from the provided users) and both (for getting tweets either mentioning or from the provided user)

The following controls are available: - Limit: Limit the perimeter of the search to the first “limit” tweets - Search: Perform the search and show the results on the screen. - Anonymised search: Perform the search and show anonymised results on the screen. User mentions and authors are replaced by a mention USER. - Anonymise: Permanently replace all matching tweets user mentions and authors by USER. - Delete: Permanently delete all the matching tweets.

The configuration page

In the configuration page, you can change settings of the tool, you can check the status of the various processes/pipelines of the tool and you can add, delete and modify topics and their associated requests, languages for geolocation and the list of the “important users” and email alert subscribers. When changing anything in the “Signal detection” or “General” sections, do not forget to click on the “Update Properties” button at the end of the “General” section. The following sections describe the configuration page in more detail.

Status

The status section enables you to quickly assess the latest time point and/or status of the processes for tweet collection (Tweet Search), and geolocation, aggregation and signal detection (Detection pipeline).

In the status section, you can tell if the embedded database, the search pipeline and the detect pipeline processes are running. You can click on “activate” or “stop” the epitweetr tasks. On Windows the tasks wim be permanently registered as scheduled tasks so they will run automatically or can be manually run from the Windows task scheduler.

Requirements & alerts pipeline

You need to run manually the tasks of dependencies, geonames and languages by clicking in the buttons “Run dependencies”, “Run geonames” and “Run languages” the first time you use epitweetr, and then only if you are downloading new versions.

Geonames and languages relate to the geolocation and language models used by epitweetr. If you would like to update them (this is not something that needs to be done regularly, more on a yearly basis or so), then you can click on “Run”. When running the ‘download dependencies’ task, epitweetr will request you to stop the ‘epitweetr database’ in order to perform the update.

The “Run alerts” button can be used to force the start of this task in case there is any error or issue. You can check their status in the “Requirements & alerts pipeline” table.

The ‘Requirements & alerts pipeline’ table gives more information about the status of the processes of epitweetr. This is useful for troubleshooting any issues arising and for monitoring the progress. It contains the five tasks that are running in the background. GeoNames and languages are tasks that will download and update the local copies of these. This will only be triggered if we add a language or update GeoNames. The start and end dates will generally be much older than those of geotag, aggregate and alerts.

Alerts dates should be more recent if the ‘data collection & processing’ and ‘requirements & alerts’ pipelines are active and running. These are scheduled according to the detect span. The status can include running, scheduled, pending, failed or aborted (if it has failed more than three times).

Signal detection

In the signal detection section in the configuration page, you can set the signal false positive rate alpha parameter, which increases (if larger) the the detection interval (more signals are detected), or decreases (if smaller) the detection interval (fewer signals are detected).

The outlier false positive rate relates to the false positive rate for determining what an outlier is when downweighting previous outliers/signals. The lower the value, the fewer previous outliers will potentially be included. A higher value will potentially include more previous outliers.

The outlier downweight strength determines how much an outlier will be downweighted by. The higher the value the greater the downweighting. For more information please see Annex I.

epitweetr calculates a threshold to determine if the current number of tweets for a given 24-hour window exceeds what is expected (see section “How does it work? General architecture behind epitweetr > Signal detection”). This threshold is based on a default of the previous 7 days. In the “default days in baseline” field, you can change the number of days.

You can also change the default of using the previous 7 days for calculating a baseline to the previous 7 same days of the week, in order to avoid a “day of the week effect” (it may be that there are always more tweets about this topic on a Monday, for example, which could affect the signal detection).

You can also specify if the signal detection is carried out just with tweet text, or includes retweets/quotes (check the box “Default with retweets/quotes”).

The last checkbox “Default with Bonferroni correction”, take multiple testing into account, which can result in false positives. If this box is checked then the signal detection alpha parameter is divided by the number of geographical locations in which signal detection is carried out. For example, at country level, the alpha parameter is divided by the total number of countries. At continent level, the alpha parameter is divided by the total number of continents.

When changing anything in the “Signal detection” section, do not forget to click on the “Update Properties” button at the end of the “General” section.

General

• In Data directory you can view the directory that epitweetr uses to store the tweet and associated data collected. This is also the directory that the dashboard uses to obtain the datasets for displaying the visualisations. You need to set this folder when you launch epitweetr or set the environment variable ‘EPI_HOME’.

• The Search span relates to how long a search plan is carried out. The default is 60 minutes. This value controls the size of the search window of tweets. If you reduce this value you will get tweets sooner, but you may ‘waste’ requests on topics with very few tweets. If you increase its value you will take more time to get the tweets but you will get more requests for popular tweets increasing the chances of being exhaustive. You can see when you are not able to collect the tweets on the Shiny configuration page if you have more than one active plan for some topics.

• The Detect span relates to how frequently the processes of the detection pipeline (geotagging, aggregation and alert detection) are carried out. The default is 90 minutes. Email alerts are sent at the end of the detect loop. This value is treated as a lower bound, the detect loop could take more time to finish depending on the volume of tweeets and your system specifications.

• The Launch slots for the detection pipeline processes will be spaced out according to the “Detect span”, with the first one starting at midnight. These values can be used in the subscribers file of the configuration page.

• To avoid storing Twitter credentials in plain files, epitweetr uses a system dependent password store functionality, which is stored in the Password store. Depending on your system you can choose the mechanism that suits the environment where epitweetr is running. For details on each implementation see https://CRAN.R-project.org/package=keyring

  • wincred: (Windows only) uses the windows credential manager.

  • macos: (MAC only) uses the Mac OS keychain services

  • file: Uses password protected encrypted files

  • secret service: (Linux only) uses Linux secret service

  • environment: Uses environment variables (extra setup needed, see https://CRAN.R-project.org/package=keyring)

• Spark cores and spark memory: The memory allocation for epitweetr in terms of CPU (Spark cores) and RAM (Spark Memory) is also defined in the “general” section. The default is 6 cores and 6 GB of RAM. This will depend on the CPU and RAM capacity of your machine and it has to be equal or less than that.

• Geolocation threshold: During the geolocation process, sets of words are processed and potential matches to existing locations are determined and given a score. The higher the score, the greater the probability that the geolocation is correct. A threshold is set in epitweetr, under which any matches are not considered good enough for geolocation. The scale goes from 1 to 10, and the default is set at 5.

• Geonames URL: The URL used to download the GeoNames database (used for generating locations) is in the general section. Should this URL ever change, you can make the amendment here.

• Simplified geonames: As GeoNames is a very large file, a simplified version of it is used by default, including only existing geographical locations where the population is known. You can uncheck this option if you wish to use the whole GeoNames database.

• Stream database results: Whether to store files on disk instead of performing streaming evaluation of aggregated data frames coming from the epitweetr database. If loading the dashboard takes too long, try selecting/unselecting this.

• Maven repository: This is the URL of the maven repository that will be used to download the JAR dependencies for the detect loop, mainly Spark and Lucene.

• Winutils URL: This is the URL that will be used to download winutils.exe. This is a Windows binary necessary for running Spark locally on Windows. If you do not want to use this version you can produce it yourself by downloading Hadoop 2.8.4 or higher and compiling it on a Windows machine.

• Region disclaimer: If you would like to add a disclaimer to the map you are using. This disclaimer is added to the image export of the dashboard map, and also the PDF export of the dashboard.

Twitter authentication

You have two options for authentication to collect tweets, using a Twitter account (utilising the rtweet package) and using a Twitter developer application. You can select which option you will use in the Twitter authentication section. User can choose the Twitter apis to use.

The admin email will receive epitweetr notifications of regular sanity checks of the application indicating, for example, if any of the tasks/pipelines has failed.

See section “How does it work? General architecture behind epitweetr > Collection of tweets > Twitter authentication” for more details on how to do the Twitter authentication.

Email authentication (SMTP)

In this section, you need to specify the email authentication (SMTP) details for the email that will send the alerts.

If Unsafe certificates is checked, then epitweetr will use your SMTP server even if the server sends an invalid certificate.

When changing anything in the “general” section, do not forget to click on the “Update Properties” button.

Topics

Topics are what determines what tweets epitweetr collects. This is done via an Excel spreadsheet that contains the topics and the associated requests that epitweetr uses to query Twitter API with.

A query consists of keywords and operators that are used to match on tweet attributes. See section “How does it work? General architecture behind epitweetr > Collection of tweets > Topics of tweets to collect and queries” for more details about queries.

epitweetr comes with a default list of topics as used by the ECDC Epidemic Intelligence team at the date of package generation (1^st of September, 2020). You can download this list of topics and upload your own in the “Available Topics” section in the configuration page. See section “How does it work? General architecture behind epitweetr > Collection of tweets > Topics of tweets to collect and queries” for more details on how to structure the topics list.

In the topics section on the configuration page, you can view the topic, the associated query, the query length and how many active search plans are associated with the query. If more than one search plan is active, this means that epitweetr did not manage to collect all possible tweets in the last session. Additionally, you can see the progress and the number of requests from the last search plan.

Languages

In the languages section, you can determine which language models are used to identify text during the geolocation process. The default languages are French, English, Portuguese and Spanish. You can download and upload the language models in the “Available Languages” section and add and delete languages used by epitweetr in the “Active Languages” section. Please consider the computational cost of adding too many languages, depending on the capacity of your machine.

All default languages (English, French, Spanish and Portuguese) must be downloaded before adding new languages or deleting any of the default languages.

The troubleshoot page

The troubleshoot page has two functionalities “Create a snapshot file” and “Diagnostics”

Downloading outputs from the interactive user interface (Shiny app)

Each visualisation on the Shiny app dashboard can be downloaded as an image, using the “image button”. A png is a portable network graphic file and is a versatile file format for images that do not need to be of a very high resolution (e.g. professional print graphics).

Note that the png format is not supported in the Internet Explorer browser (but you can download a svg file instead).

You can also download the data of each visualisation by clicking on the data button. This will give you a csv file containing the underlying data that you can use for further analysis or to create your own graphs.

Alternatively, you can use the PDF or the Md button at the bottom of the filters to download a PDF or an HTML file of the dashboard Note that for this you will need to have MiKTeX or TinyTeX installed.

EARS as a Linear Model

We first observe that the above estimation of \(\mu\) and \(\sigma^2\) through \(\overline{y}_0\) and \(s_0^2\) at time 0 can be embedded within a linear regression model, i.e. for \(i=1, \ldots, n\) we model \[ y_i = \mu + \epsilon_i, \quad\text{where}\quad \epsilon_t \stackrel{\text{iid}}{\sim} N(0, \sigma^2). \] Note that we, for compatibility with the standard exposition in linear model theory, have indexed the \(y\) values s.t. \(y_{-n}\) corresponds to \(y_1\) and \(y_{-1}\) corresponds to \(y_n\). In matrix terms let \(\mathbf{y} = (y_{1},\ldots,y_n)'\) and for the intercept-only model the design matrix is \(\mathbf{X} = (1,\ldots,1)'\), which has rank \(k=1\). Thus from standard OLS theory: \[ \begin{align*} \hat{\mu} &= (\mathbf{X}'\mathbf{X})^{-1} \mathbf{X}' \mathbf{y} = \frac{1}{n}\sum_{i=1}^n y_i, \end{align*} \] which corresponds to \(\overline{y}_0\). Furthermore, let the raw residuals be defined as \(e_i = y_i - \hat{\mu}\) for \(i=1,\ldots, n\) and denote by \(\mathbf{e}=(e_1,\ldots,e_n)'\) the corresponding vector of residuals. Then \[ \mathbf{e} = \mathbf{y} - \hat{\mathbf{y}} = \mathbf{y} - \mathbf{P} \mathbf{y} = (\mathbf{I}-\mathbf{P}) \mathbf{y} \] where \(\mathbf{P} = \mathbf{X} (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\) is the so called hat-matrix known from linear modelling. With this notation we can write up the estimate for \(\sigma^2\) as in Chatterjee and Hadi (1988):

\[ \hat{\sigma}^2 = \frac{\mathbf{e}' \mathbf{e}}{n-k} = \frac{\mathbf{y}'(\mathbf{I}-\mathbf{P})\mathbf{y}}{n-k} = \frac{1}{n-1} \sum_{t=-7}^1 (y_t - \hat{\mu})^2, \] which corresponds to the above used expression for \(s_0^2\).

Downweighting

We now compute the so called externally Studentized residuals (Chatterjee and Hadi 1988) \[ r_i^* = \frac{e_i}{\hat{\sigma}_{(i)}\sqrt{1-p_{ii}}}, \quad i=1, \ldots, n, \] where \(p_{ii}\) is i’th diagonal element of the hat-matrix \(\mathbf{P}\) from the corresponding linear model used above. Furthermore, \[ \hat{\sigma}_{(i)}^2 = \frac{\mathbf{y}_{(i)}' (\mathbf{I}-\mathbf{P}_{(i)}) \mathbf{y}_{(i)}}{n-k-1} \] is the variance estimate obtained from a linear regression, where the i’th observation is removed. Linear modelling theory (Chatterjee and Hadi 1988) now states that \[ r_i^* \stackrel{\text{identical}}{\sim} t(n-k-1). \] Note that the residuals are only identically distributed, because they are not independent (see Section 4.2.1. of Chatterjee and Hadi (1988) for details). However, the above distributional form allows us to assess for each historic value, if it can be considered as an outlier. For this purpose define \(r_{\text{threshold}}\) as the \(1-\alpha_{\text{outlier}}\) quantile of the t-distribution with \(n-k-1\) degrees of freedom. A historic value is an outlier (for which one possible explanation is that it originates from a true increase in tweets, e.g. an outbreak situation), if \(r_i^* > r_{\text{threshold}}\). We shall use this to formulate a weighting schemes for the historic values:

Downweight-Outliers: \[ \begin{align} w^{(\text{dw})}_i &= \left\{ \begin{array}{ll} 1 & \text{if } r_i^* < r_{\text{threshold}}\\ \left(\frac{r_{\text{threshold}}}{(r_i^*)}\right)^k & \text{otherwise} \end{array} \right. \\ &= \min\left\{1,\left(\frac{r_{\text{threshold}}}{r_i^*}\right)^k\right\}, \end{align} \] where the decay parameter \(k>0\) is a known quantity. In the original Farrington et al. (1996) algorithm, \(k=2\) was used. Furthermore, a threshold value of 1 was used. In the later Noufaily et al. (2013) paper, however, a threshold value of 2.58 was recommended. Note: both values are for the standardized Anscombe residuals, which follow a standard normal distribution. If we take corresponding quantiles for the t-distribution with 6 degrees of freedom the values would be 1.09 and 3.72. Note also that the term \((r_{\text{threshold}}/r_i^*)^k\) is a slight adaptation of Farrington et al. (1996), which instead uses \(1/(r_i^*)^2\). The advantage of our proposal is that it ensures a smooth handling of values around the threshold if the threshold is not 1. It might be worth considering a higher power than 2 to ensure an even larger down-weighting for gross outliers. The current default value for the decay parameter in epitweetr is 4.

Finally, as in Farrington et al. (1996), we normalise the weights such that they yield a sum of \(n\) by \[ w_i^* = n \times \frac{w_i}{\sum_{i=1}^n w_i} \] and then re-fit the linear model with these weights. For this purpose define the weight matrix as \(\mathbf{W} = \operatorname{diag}(w_1^*,\ldots,w_n^*)\). We can use a subsequent weighted least squares approach to find \[ \begin{align*} \hat{\mu}_W &= (\mathbf{X}' \mathbf{W} \mathbf{X})^{-1} \mathbf{X}' \mathbf{W} \mathbf{y} = \frac{1}{n}\sum_{i=1}^n w_i^* y_i, \end{align*} \] where the 2nd equal sign is because \((\mathbf{X}' \mathbf{W} \mathbf{X})=\sum_{i=1}^n w_i=n\) and \(\mathbf{X}' \mathbf{W} \mathbf{y} = \sum_{i=1}^n w_i^* y_i\). Furthermore, \[ s_W^2 = \frac{\mathbf{y}'(\mathbf{I}-\mathbf{P}_W)\mathbf{y}}{n-k} = \frac{\sum_{i=1}^n w_i^*(y_i - \mu_W)^2}{n-1}, \] where \(P_W = \mathbf{X} (\mathbf{X}'\mathbf{W} \mathbf{X})^{-1}\mathbf{X}\mathbf{W}\) is the hat-matrix of the weighted least squares.

The downweighted procedure thus operates with \(\mu_{W}\) and \(s_W^2\) instead of \(\overline{y}_0\) and \(s_0^2\), respectively, when computing the upper limit \(U_0\) using the above mentioned formula.

Example of the downweighting approach using Ebola data

Figure 5 below shows the upper limit of the signal detection threshold for epitweetr Ebola data, both the original (in red) with no downweighting and the downweighted upper threshold (in blue) after taking previous signals in the historic values into account. Note that the downweighted upper threshold detects three additional signals, compared to the original threshold.

The troubleshoot page

After running the diagnostics in the troubleshoot page, you can see the checks and status on the following aspects:

• scheduler: R package taskscheduleR is installed. Only applicable for Windows machines
• twitter_auth: Twitter token has been created after authenticating either with Twitter account or Twitter developer application
• search_running: search task is running
• database_running: embedded database is running
• tweets: tweets have been collected
• os64: R is 64bits
• java: Java is installed and accessible to epitweetr
• java64: Java 64bits is installed and accessible to epitweetr
• java_version: the Java version installed is compatible with epitweetr
• winmsvc: Microsoft Visual C++ 2010 SP1 Redistributable Package is installed. Only applicable for Windows machines
• detect_activation: detect loop has been activated
• detection_running: detection task is running
• winutils: winutils is installed. If false, it can be downloaded by running the update dependencies task. Only applicable for Windows machines
• java_deps: java dependencies are installed
• move_from_temp: epitweetr can atomically move files from temporary folder to data directory
• tar_gz: checks if tar.gz can be built, which is necessary for creating a compressed snapshot.
• geonames: Geonames.org database is downloaded and indexed
• languages: languages vectors are downloaded and indexed
• aggregate: aggregate task has successfully run
• alerts: alerts have been created
• pandoc: Pandoc is installed and accessible to epitweetr. Necessary for PDF creation.
• tex: a tex distribution is installed and accessible to epitweetr. Necessary for PDF creation.

Management of epitweetr pipelines (‘epitweetr database’, ‘Data collection & processing’, ‘Requirements & alerts’) in Windows

After activating the three pipelines (‘epitweetr database’, ‘Data collection & processing’, ‘Requirements & alerts’) from the configuration page of epitweetr (Windows), three tasks will be created in the task scheduler and three terminal windows will be prompted. Please note that if the computer is logged/turned off or the terminal windows are closed, the pipelines will stop.

If you activate these tasks from the configuration page of epitweetr again, the system will overwrite the tasks created in the task scheduler. Instead, after the first successful activation of these tasks from epitweetr, you can easily manage these from the task scheduler. You can stop these tasks by ending and disabling the tasks in the task scheduler, and you can restart these tasks by enabling and running these in the task scheduler.

You can also force stopping these tasks in the configuration page of epitweetr by clicking on ‘stop’.

In the task scheduler, you can establish that the tasks “run whether the user is logged on or not” to avoid that these tasks stop when you log off or restart the computer. In this case, you may not see the prompted terminal windows when the tasks are running.

Management of epitweetr pipelines (‘epitweetr database’, ‘Data collection & processing’, ‘Requirements & alerts’) in Linux and Mac

Since the three pipelines (‘epitweetr database’, ‘Data collection & processing’, ‘Requirements & alerts’) in Linux or Mac have to be run manually, if the computer is logged/turned off or the terminal windows are closed, the pipelines will stop. Please remember to follow the steps in the section Setting up tweet collection and the alert detection loop to run these tasks again.

The pipelines can be also stopped in the configuration page of epitweetr by clicking on ‘stop’.

Running ‘epitweetr database’, ‘Data collection and processing’ and ‘Requirements and alerts’ pipelines

Change the user of the Twitter authentication when using a Twitter account

1. End and disable the ‘Data collection & processing’ and ‘epitweetr database’ pipelines in the task scheduler (Windows), close the R/terminal window with the pipelines or forcing the pipelines in the configuration page (Windows, Linux and Mac)

2. Search for a file called “.rtweet_token” in hidden files. It is usually saved in the Documents folder.

3. Delete that file.

4. Click on “Update properties” in the configuration page of epitweetr.

5. Enable and run the pipelines in the task scheduler or active them in the configuration page (Windows), or run the command in a new R/terminal window with the pipelines (Windows, Linux and Mac). More details are available in the section “Setting up tweet collection and the alert detection loop”

Downloading GeoNames and/or languages

The “Launch slots” in the configuration page show NAs instead of the time slots

If it is the first time that you install and launch epitweetr, the geotag task of the detection pipeline has to be run at least once in order to see time slots in the “Launch slots” in the configuration page.

Downloading PDF of the dashboard

Different totals in dashboard outputs

When counting the total tweets in the dashboard of the Shiny app or in the downloadable data, you might get differences in the total numbers of tweets between the three outputs. This might be due to the following reasons:

1. World (all) versus World (geolocated)

   • The default option for the regions in World (all), this means that also non-geolocated tweets are included in the trendline, but only geolocated tweets can be visualized in the maps and the most frequent words figure, therefore the overall total of tweets can differ between these outputs wen selecting World (all) or the empty default.

2. Country specific analysis

   • If you select only one country in the filters, the trendline will show all tweets for this country, but the map will show the tweets on a subnational level in the map. It could be that some tweets might have been geolocated to a certain country, but without further subnational data. These tweets will then be visible in the trendline total, but not in the subnational bubbles in the map.

3. Most frequent words/hashtags

   • In contrast to the other outputs in the dashboard, the most frequent words figure is always based on tweet location regardless of the filter (due to memory capacity). Therefore, if user location or both locations are selected in the location filter, this figure might have a different total then the other two outputs.

Receiving only real-time alerts

This relates to users who have selected topics and/or regions for receiving related alerts in real-time or have selected topics and/or regions for receiving related alerts on a scheduled time span. If, in these cases, you only receive real-time alerts with all topics and regions, it may be that no time slots have been included in the subscribers file from the configuration page. These time slots are used for the scheduled alerts and if no slots are included in the file, alerts from all topics and regions are sent as real-time alerts.

Not receiving email alerts

If you do not receive email alerts and you see an error in epitweetr referring to denied login, it means that epitweetr could not login to the email account provided in the configuration page. Some of the reasons for that are:

• The server or port included in the configuration page are incorrect
• The attempt of epitweetr to login to the email account is being blocked by the server. This can happen with some organisational email accounts. In that case, please contact the IT department of your organisation
• If using Gmail account, you need to allow less secure apps in the settings of your account