mRpostman basics

Introduction

mRpostman is an easy-to-use IMAP client that provides tools for message searching, selective fetching of message attributes, mailbox management, attachment extraction, and several other IMAP features, paving the way for email data analysis in R. To do so, this package makes extensive use of the {curl} package and the libcurl C library.

In this vignette, we present all available methods and functions of this package, but not all the possibilities one can explore.

IMPORTANT:

1. In version 0.9.0.0, mRpostman went trough substantial changes, including ones that have no backward compatibility with versions <= 0.3.1. A detailed vignette on how to migrate your mRpostman’s deprecated code to the new syntax is available at “Migrating old code to the new mRpostman’s syntax”.

2. Old versions of the libcurl C library ({curl}’s main engine) will cause malfunctioning of this package. If your libcurl’s version is above 7.58.0, you should be fine. If you intend to use OAuth 2.0 authentication, then you will need libcurl >= 7.65.0. To know more about the OAuth 2.0 authentication in this package, refer to the “Using IMAP OAuth2.0 authentication in mRpostman” vignette.

3. Most mail providers discontinued less secure apps access. If it is still available and you are comfortable with this type of access you can enable this option for your account on your mail provider. Some providers, such as Yahoo Mail, also offer the option to generate password to be used by third-party apps such as mRpostman. The other option, as mentioned above, is to set up OAuth2 (two-factor authentication) in order to access your mailbox. Please also refer to the “Using IMAP OAuth2.0 authentication in mRpostman” vignette.

Providers and their IMAP urls

* For Office 365 accounts, the username should be set as user@yourcompany.com or user@youruniversity.edu for example.

Package Structure

The package is now implemented under a OO framework, using an R6 class. The main functionalities of mRpostman are implemented as methods of the R6 class called ImapCon. There are also a few independent functions. All methods and functions are described below:

• configuration methods: configure_imap(), reset_url(), reset_username(), reset_password(), reset_verbose(), reset_buffersize(), reset_timeout_ms(), reset_xoauth2_bearer();
• server-capabilities method: list_server_capabilities();
• mailbox-operations methods: list_mail_folders(), select_folder(), examine_folder(), rename_folder(), create_folder(), list_flags();
• single-search methods: search_before(), search_since(), search_period(), search_on(), search_sent_before(),search_sent_since(), search_sent_period(), search_sent_on(), search_string(), search_flag(), search_smaller_than(), search_larger_than(), search_younger_than(), search_older_than();
• the custom-search method and its helper functions: search();
  • relational operators functions: AND(), OR();
  • criteria definition functions: before(), since(), on(), sent_before(), sent_since(), sent_on(), string(), flag(), smaller_than(), larger_than(), younger_than(), older_than();
• fetch methods: fetch_body(), fetch_header(), fetch_text(), fetch_metadata(), fetch_attachments_list(), fetch_attachments();
• attachments methods: list_attachments(), get_attachments(), fetch_attachments_list(), fetch_attachments();
• complementary methods: copy_msg(), move_msg(), esearch_min_id(), esearch_max_id(), esearch_count_msg(), delete_msg(), expunge(), add_flags(), remove_flags(), replace_flags().

1) How do I start? (Connection configuration)

After setting the authentication method in your mail provider, you have to configure an IMAP connection:

library(mRpostman)

# IMAP settings

# Outlook - Office 365
con <- configure_imap(
  url="imaps://outlook.office365.com",
  username="user@your_company.com",
  password=rstudioapi::askForPassword()
  )

# Gmail
con <- configure_imap(
  url = "imaps://imap.gmail.com",
  username = "user",
  password = rstudioapi::askForPassword()
  )

# Hotmail
con <- configure_imap(
  url = "imaps://imap-mail.outlook.com",
  username = "user@hotmail.com",
  password = rstudioapi::askForPassword()
  )

# Yahoo Mail
con <- configure_imap(
  url="imaps://imap.mail.yahoo.com/",
  username="your_user",
  password=rstudioapi::askForPassword()
  )

# AOL Mail
con <- configure_imap(
  url="imaps://export.imap.aol.com/",
  username="your_user",
  password=rstudioapi::askForPassword()
  )

# Yandex Mail
con <- configure_imap(
  url="imaps://imap.yandex.com",
  username="your_user",
  password=rstudioapi::askForPassword()
  )

# ... and any other mail provider with IMAP support

Other useful options are: timeout_ms, verbose = TRUE, buffersize. Further {curl} options related to IMAP functionalities can be passed to configure_imap(), but you probably won’t need it. See curl::curl_options().

Since version 0.9.0.0, this package provides more flexibility to the user in the sense that you can modify the connection parameters for specific commands or parts of a script, using the reset_*() methods. This prevents users from having to call configure_imap() multiple times during a session or in a script. It is particularly useful when the user is going to perform some fetch operation, for example. In this case, it is recommended to increase the timeout_ms and set verbose = FALSE.

The con object that we created in the example above has the ImapCon R6 class. Now, almost 99% of the other IMAP commands to be performed on the server will be called following the structure: con$method(). The exceptions are the list_attachments() function and the helper functions of the custom-search group.

As you will see, the R6 framework combined with {curl} will make this package works such a session-based IMAP client. Besides this, for some commands, users will be able to use the tidy approach with pipes. All this together provides an elegant way of accessing your IMAP provider, searching and fetching emails, and managing your mailbox as well.

2) Server capabilities

Once the connection is configured, it is important to know which capabilities your IMAP provider offers to users. This impacts on which type of operations you are allowed to perform. For example, if your server has the WITHIN extension you can use the WITHIN search methods search_younger_than() and search_older_than(); if the server has the ESEARCH capability, besides being allowed to use the esearch_*() methods, you can optimize all your search functions with the esearch = TRUE parameter; if you see the MOVE capability, then you can use the move_msg() method. Therefore, to know all your server capabilities, you can use list_server_capabliltites.

con$list_server_capabilities()

3) Mailbox commands

con$list_mail_folders()

con$create_folder(name = "New Folder")

con$select_folder(name = "INBOX")

con$select_folder(name = "Inbox")

con$examine_folder()

con$examine_folder(name = "Sent")

con$select_folder(name = "CRAN Messages")

con$rename_folder(new_name = "CRAN")

con$rename_folder(name = "Sent", new_name = "Sent2")

con$list_flags()

4) Single-search

All search methods will return a numeric vector containing the results from the search. This allows users to chain fetch operations together with search one. You can also NEGATE all search expressions by setting negate = TRUE.

If your server supports ESEARCH, we recommend you to use it. It will prevent your results from being truncated when there are too many messages ids and you didn’t set a high buffersize in confiure_imap()¹. With “ESEARCH”, the results will be condensed to groups of sequences similar to what does. For instance, if your search returns 10000 results, it is better to have condensed groups such as 1:10, 12, 23:27, ... instead of a sequence of 1, 2, 3, 4, 5, 6, ..., 10, 12, 23, 24, 25, .... If you can’t use ESEARCH, or if your results are being truncated even with ESEARCH, you can try to increase your buffersize in configure_imap() to avoid this.

con$select_folder(name = "INBOX")

res <- con$search_before(date_char = "07-Sep-2020")

res

res <- con$search_before(date_char = "07-Sep-2020",
                         use_uid = TRUE,
                         flag = c("ANSWERED", "SEEN"))

res

res <- con$search_before(date_char = "07-Sep-2020",
                         use_uid = TRUE,
                         flag = c("ANSWERED", "SEEN"),
                         esearch = TRUE)

res

res <- con$search_before(date_char = "07-Sep-2020",
                         negate = TRUE,
                         use_uid = TRUE)
                         
res

res <- con$search_since(date_char = "07-Sep-2020",
                        use_uid = TRUE)
                         
res

res <- con$search_period(since_date_char = "02-Jan-2020",
                         before_date_char = "30-Jun-2020")
                         
res

res <- con$search_period(since_date_char = "02-Jan-2020",
                         before_date_char = "30-Jun-2020",
                         negate = TRUE)
                         
res

con$search_on(date_char = "02-Jan-2020")

con$search_sent_before(date_char = "07-Sep-2020")

res <- con$search_sent_before(date_char = "07-Sep-2020",
                              negate = TRUE,
                              use_uid = TRUE,
                              flag = c("ANSWERED", "SEEN"))

res

con$search_sent_since(date_char = "07-Sep-2020")

con$search_sent_period(since_date_char = "02-Jan-2020",
                       before_date_char = "30-Jun-2020")

con$search_sent_on(date_char = "30-Jun-2020")

con$select_folder(name = "K-State")

con$search_string(expr = "xpto@k-state.com", where = "TO")

con$search_string(expr = "xpto@k-state.edu", where = "FROM")

con$search_string(expr = "PhD offer", where = "SUBJECT")

con$search_string(expr = "Dear Allan" where = "TEXT")

con$search_string(expr = "Dear Allan" where = "BODY")

con$search_flag(name = c("ANSWERED", "Seen"), use_uid = TRUE)

con$search_smaller_than(size = 512000) # smaller than 512KB

con$search_larger_than(size = 512000) # larger than 512KB

con$search_younger_than(seconds = 3600) # msgs received less than one hour (3600 sec)

con$search_older_than(seconds = 3600) # msgs received more than one hour ago (3600 sec)

5) Custom-search

The search() method and its helper functions enable users to create a vast number of complex and customized search requests by combining different criteria, using all the types of searches previously presented in this document.

These are the helper functions you can use inside search():

1. Relational operators: AND(), and OR();
2. Criteria definition: before(), since(), on(), sent_before(), sent_since(), sent_on(), string(), flag(), smaller_than(), larger_than(), younger_than(), and older_than().

NOTE: IMAP queries follows Polish notation, i.e. operators such as OR come before arguments, e.g. “OR argument1 argument2”. Therefore, the relational operators functions in this package should be used like the following examples: OR(before(date_char = "17-Apr-2015"), string(expr = "Jimmy", where = "FROM")). Even though there is no “AND” operator in the IMAP protocol, this package adds a helper function AND() to indicate multiple arguments that must be searched together, e.g. AND(since(date_char = "01-Jul-2018"), larger_than(size = 16000)).

Example 1: Searching for messages (in “INBOX”) containing the string “Kansas State University” in the “SUBJECT” header field AND that were received before “02-Jan-2020”.

con$select_folder(name = "INBOX")


res <- con$search(request = AND(string(expr = "Kansas State University", where = "SUBJECT"),
                                before(date_char = "02-Jan-2020")))

res

Example 2: Searching (using UID) for messages received from “@k-state.edu” OR “@ksu.edu”.

con$search(request = OR(string(expr = "@k-state.edu", where = "FROM"),
                        string(expr = "@ksu.edu", where = "FROM")),
           use_uid = TRUE)

6) Fetch

You can fetch the full content of messages, or their parts, such as the header, text, or specific metadata fields. Besides this, you can also fetch a message attachments list or the attachment files themselves, downloading them to the disk.

We usually fetch messages after a search operation. Given the output of the search functions in mRpostman, you can use the pipe %>% to chain the search and the fetch operations together. Using the base R approach is perfectly possible as well.

In the main fetch methods (those that are not related to attachment fetching), you can choose to write the fetch results to disk (working directory) using write_to_disk = TRUE. If you opt to do so, mRpostman saves the fetched content to a .txt file in the following folder structure: working directory > imap.server.url > mail folder name. The text files will be named after the id of the fetched message. If the operation was executed using the UID, the “UID” prefix is added to the file names.

IMPORTANT:

1. If the fetch operation is to be chained after a search, the use_uid arguments in the two operations have to be the same. Otherwise, an error will occur or the fetch will be performed on wrong messages’ ids.

2. It is always recommended to increase the timeout_ms before fetch_body(), fetch_text(), and fetch_attachents() operations as sometimes the operation may hang for a few seconds while fetching the message parts.

3. if you have configured a connection with verbose = TRUE, it is extremely recommended that you reset it to FALSE before a fetching operation. The verbose = TRUE option fill the console with the whole flux of information between the server and the client, drastically slowing the speed of the process and your R session.

# increasing timeout_ms
con$reset_timeout_ms(x = 30000) # ... to 30 secs

# and supposing that you had verbose = TRUE before
con$reset_verbose(x = FALSE)

# tidy approach
con$search_string(expr = "@k-state.edu", where = "FROM") %>%
  con$fetch_body(write_to_disk = TRUE, keep_in_mem = FALSE)

# ---------------

# base R approach
res <- con$search_string(expr = "@k-state.edu", where = "FROM")

con$fetch_body(msg_id = res, write_to_disk = TRUE, keep_in_mem = FALSE)

# tidy approach
out <- con$search_since(date_char = "15-Aug-2019", use_uid = TRUE) %>%
  con$fetch_header(use_uid = TRUE, fields = c("DATE", "SUBJECT"))

out

# ---------------

# base R approach
res <- con$search_since(date_char = "15-Aug-2019", use_uid = TRUE)

out <- con$fetch_header(use_uid = TRUE, fields = c("DATE", "SUBJECT"))

out

con$search_since(date_char = "15-Aug-2019") %>%
  con$fetch_text(write_to_disk = TRUE, keep_in_mem = FALSE)

out <- con$search_on(date_char = "15-Aug-2019", use_uid = TRUE) %>%
  con$fetch_metadata(use_uid = TRUE, attribute = c("INTERNALDATE", "UID", "ENVELOPE"))

7) Attachments

Previously, mRpostman had two attachment-related functions: list_attachments() and get_attachments(). Both were to be used after a fetching operation because they would use the content fetched in a previous step to parse the attachments list and/or decode the content to files. These methods are still available in the current version of the package. However, two more methods were implemented since version 0.9.0.0: fetch_attachments_list() and fetch_attachments(). These methods are not contingent on a previous fetch operation, since they are a properly fetch operation.

Let’s take a look at the first two attachments functionalities available in mRpostman.

# using fetch_body()
con$search_since(date_char = "23-Sep-2019") %>%
  con$fetch_body(write_to_disk = TRUE) %>%
  list_attachments() # don't need to specify the connection object here

# ------------------

# using fetch_text()
con$search_since(date_char = "23-Sep-2019") %>%
  con$fetch_text(write_to_disk = TRUE) %>%
  list_attachments() # don't need to specify the connection object here

# using fetch_body()
con$search_since(date_char = "23-Sep-2019") %>%
  con$fetch_body(write_to_disk = TRUE) %>%
  con$get_attachments()

# ------------------

# using fetch_text()
con$search_since(date_char = "23-Sep-2019") %>%
  con$fetch_text(write_to_disk = TRUE) %>%
  con$get_attachments()

con$search_since(date_char = "23-Sep-2019") %>%
  con$fetch_attachments_list()

con$search_since(date_char = "23-Sep-2019") %>%
  con$fetch_attachments()

8) Complementary operations

Here we present other functions to perform very useful complementary IMAP operations.

con$select_folder(name = "INBOX")

con$search_since(date_char = "10-may-2019") %>%
  con$copy_msg(to_folder = "K-State")

con$esearch_min_id(flag = c("Answered", "Seen"))

con$esearch_min_id(flag = c("Answered", "Seen"))

con$esearch_count(flag = c("Answered", "Seen"))

con$select_folder(name = "Trash")

con$search_before(date_char = "10-may-2012") %>%
  con$delete_msg()

con$delete_msg(msg_id = 66128)

# expunge the entire mail folder
con$expunge()

# expunge selected msg UID
con$delete_msg(msg_id = 71171, use_uid = TRUE) %>%
  expunge()

con$select_folder(name = "INBOX")

con$search_since(date_char = "01-Sep-2020", use_uid = TRUE) %>%
  con$add_flags(flags_to_set = "\\Answered", use_uid = TRUE)

con$search_since(date_char = "01-Sep-2020", use_uid = TRUE) %>%
  con$replace_flags(flags_to_set = c("\\Seen", "\\Flagged", use_uid = TRUE)

con$search_since(date_char = "01-Sep-2020", use_uid = TRUE) %>%
  con$remobe_flags(flags_to_unset = c("\\Seen", "\\Flagged", use_uid = TRUE)

con$search_on(date_char = "07-Sep-2020") %>%
  con$move_msg(to_folder = "K-State")

con$search_on(date_char = "07-Sep-2020") %>%
  con$copy_msg(to_folder = "K-State", reselect = FALSE) %>%
  con$add_flags(flags_to_set = "\\Deleted") %>%
  con$expunge()

References

Babcock, N. (2016), Introduction to IMAP, Blog, May 2016, http.

Crispin, M. (2003), INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1, RFC 3501, March 2003, http.

Freed, N. and Borenstein, N. (1996), Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types, RFC 2046, November 1996, http.

Gungor, A. (2018), Using IMAP Internal Date for Forensic Email Authentication, Articles, Forensic Focus, http.

Heinlein, P. and Hartleben, P. (2008). The Book of IMAP: Building a Mail Server with Courier and Cyrus. No Starch Press. ISBN 978-1-59327-177-0.

Resnick, P. (2001), Internet Message Format, RFC 2822, April 2001, http.

Resnick, P. (2008), Internet Message Format, RFC 5322, October 2008, http.

Ooms, J. (2020), curl: A Modern and Flexible Web Client for R. R package version 4.3, http.

Stenberg, D. Libcurl - The Multiprotocol File Transfer Library, http