aRxiv tutorial

Search terms

Generally, one would ignore id_list and focus on forming the query argument. The aRxiv package includes a dataset query_terms that lists the terms (like au) that you can use.

query_terms

##               term                                      description
## 1               ti                                            Title
## 2               au                                           Author
## 3              abs                                         Abstract
## 4               co                                          Comment
## 5               jr                                Journal Reference
## 6              cat                                 Subject Category
## 7               rn                                    Report Number
## 8              all                                 All of the above
## 9    submittedDate Date/time of initial submission, as YYYYMMDDHHMM
## 10 lastUpdatedDate        Date/time of last update, as YYYYMMDDHHMM

Use a colon (:) to separate the query term from the actual query. Multiple queries can be combined with AND, OR, and ANDNOT. The default is OR.

arxiv_count('au:Peter au:Hall')

## [1] 38112

arxiv_count('au:Peter OR au:Hall')

## [1] 38112

arxiv_count('au:Peter AND au:Hall')

## [1] 161

arxiv_count('au:Hall ANDNOT au:Peter')

## [1] 2764

It appears that in the author field (and many other fields) you must search full words, and that wild cards are not allowed.

arxiv_count('au:P* AND au:Hall')

## [1] 0

arxiv_count('au:P AND au:Hall')

## [1] 1236

arxiv_count('au:"P Hall"')

## [1] 44

Subject classifications

arXiv has a set of 155 subject classifications, searchable with the prefix cat:. The aRxiv package contains a dataset arxiv_cats containing the categories, short and long descriptions, as well as field (and, for Physics, subfield). Here are the column names.

colnames(arxiv_cats)

## [1] "category"          "field"             "subfield"         
## [4] "short_description" "long_description"

Here are the statistics categories.

arxiv_cats[arxiv_cats$field=="Statistics", c("category", "short_description")]

##     category short_description
## 150  stat.AP      Applications
## 151  stat.CO       Computation
## 152  stat.ME       Methodology
## 153  stat.ML  Machine Learning
## 154  stat.OT  Other Statistics
## 155  stat.TH Statistics Theory

To search these categories, you need to include either the full term or use the * wildcard.

arxiv_count('cat:stat')

## [1] 0

arxiv_count('cat:stat.AP')

## [1] 17806

arxiv_count('cat:stat*')

## [1] 116194

Dates and ranges of dates

The terms submittedDate (date/time of first submission) and lastUpdatedDate (date/time of last revision) are particularly useful for limiting a search with many results, so that you may combine multiple searches together, each within some window of time, to get the full results.

The date/time information is of the form YYYYMMDDHHMMSS, for example 20071018122534 for 2007-10-18 12:25:34. You can use * for a wildcard for the times. For example, to get all manuscripts with initial submission on 2007-10-18:

arxiv_count('submittedDate:20071018*')

## [1] 196

But you can’t use the wildcard within the dates.

arxiv_count('submittedDate:2007*')

## [1] 0

To get a count of all manuscripts with original submission in 2007, use a date range, like [from_date TO to_date]. (If you give a partial date, it’s treated as the earliest date/time that matches, and the range appears to be up to but not including the second date/time.)

arxiv_count('submittedDate:[2007 TO 2008]')

## [1] 55749

Metadata limitations

The arXiv metadata has a number of limitations, the key issue being that it is author-supplied and so not necessarily consistent between records.

Authors’ names may vary between records (e.g., Peter Hall vs. Peter G. Hall vs. Peter Gavin Hall vs. P Hall). Further, arXiv provides no ability to distinguish multiple individuals with the same name (c.f., ORCID).

Authors’ institutional affiliations are mostly missing. The arXiv submission form does not include an affiliation field; affiliations are entered within the author field, in parentheses. The metadata instructions may not be widely read.

There are no key words; you are stuck with searching the free text in the titles and abstracts.

Subject classifications are provided by the authors and may be incomplete or inappropriate.

Limit time between search requests

Care should be taken to avoid multiple requests to the arXiv API in a short period of time. The arXiv API user manual states:

In cases where the API needs to be called multiple times in a row, we encourage you to play nice and incorporate a 3 second delay in your code.

The aRxiv package institutes a delay between requests, with the time period for the delay configurable with the R option "aRxiv_delay" (in seconds). The default is 3 seconds.

To reduce the delay to 1 second, use:

options(aRxiv_delay=1)

Don’t do searches in parallel (e.g., via the parallel package). You may be locked out from the arXiv API.

Limit number of items returned

The arXiv API returns only complete records (including the entire abstracts); searches returning large numbers of records can be very slow.

It’s best to use arxiv_count() before arxiv_search(), so that you have a sense of how many records you will receive. If the count is large, you may wish to refine your query.

arXiv has a hard limit of around 50,000 records; for a query that matches more than 50,000 manuscripts, there is no way to receive the full results. The simplest solution to this problem is to break the query into smaller pieces, for example using slices of time, with a range of dates for submittedDate or lastUpdatedDate.

The limit argument to arxiv_search() (with default limit=10) limits the number of records to be returned. If you wish to receive more than 10 records, you must specify a larger limit (e.g., limit=100).

To avoid accidental searches that may return a very large number of records, arxiv_search() uses an R option, aRxiv_toomany (with a default of 15,000), and refuses to attempt a search that will return results above that limit.

Make requests in batches

Even for searches that return a moderate number of records (say 2,000), it may be best to make the requests in batches: Use a smaller value for the limit argument (say 100), and make multiple requests with different offsets, indicated with the start argument, for the initial record to return.

This is done automatically with the batchsize argument to arxiv_search(). A search is split into multiple calls, with no more than batchsize records to be returned by each, and then the results are combined.