Using the WorldCat Search API

Introduction

libbib’s ability to communicate to the WorldCat Search API is probably the most helpful capability of the package. As such, an entire vignette dedicated to this powerful tool is warranted.

Also of note is that documentation on how to use this API is scattered throughout the web, in different places, and sometimes even no longer available and only accessible through the internet archive. Because of this, the goal of this vignette is not only to document libbib’s worldcat_api_search function, but to provide a single location for compiling this archived documentation and providing information/examples of what the WorldCat Search API is capable, in general.

The terminology/nomenclature of the different WorldCat API offerings is fuzzy and inconsistent. The specific kind of Search API that this package offers usage of can, more specifically, be described as the ‘WorldCat SRU Search API version 1’.

There is a version 2 of this API, but version 1 hasn’t been sunsetted as of yet.

First, let’s speak of what this API is and what it is not.

What the WorldCat SRU Search API is

This API is allows a developer to search for bibliographic records that are cataloged in WorldCat.

The queries are made using the SRU standard search protocol (Search/Retrieve via URL) using a standard query syntax called CQL (Contextual Query Language). The WorldCat Search API doesn’t implement all features of CQL; this vignette will illustrate SRU/CQL only insofar as it is supported by this API.

This API is open to libraries that maintain both WorldCat Discovery and OCLC Cataloging subscriptions and needs an API key (called a WSKey) to work. A request can be made for a key (if your institution doesn’t already have one) via this link.

What the WorldCat SRU Search API is not

This API is not the OpenSearch/Basic API, which doesn’t allow field-specific searches and only supports keyword-anywhere searches.

This is also not the same as using the advanced search option on worldcat.org. That option only provides a subset of the bibliographic records in WorldCat and is a far less powerful tool.

This API is most akin to using the “expert search” option in OCLC’s FirstSearch, using the WorldCat database, but differs in that (a) searching the API is programmatically automate-able, and (b) the API allows for, still, more powerful search queries.

What libbib’s worldcat_api_search function provides

At its most basic, this function takes a SRU query and returns a data.table with most of the bibliographic metadata from the MARCXml that the API returns. We’ll see how the behavior of this function can be controlled by specifying certain function parameters.

This function also offers assistance with the SRU query syntax. Mainly, the function allows you to substitute the arcane search index codes for more human-readable equivalents, prefixed be a (US) dollar-sign. Examples of these aids will be explained later in the vignette.

An example of usage and what the function returns

Let’s look at an example of a simple query and what the returned data.table looks like.

Specifically, we’ll search for “Madame Bovary” by “Gustave Flaubert” and we’ll only show the first three results

library(libbib)      # load this package

result <- worldcat_api_search('$title="Madame Bovary" and
                                 $author="Gustave Flaubert"')

# get the column names
names(result)
#>  [1] "total_wc_results" "result_number"   "oclc"           "isbn"            
#>  [5] "issn"             "title"           "author"         "pub_date"        
#>  [9] "lang_code"        "bib_level"       "record_type"    "pub_place_code"  
#> [13] "publisher"        "leader"          "oh08"           "query"

# show the first three results
result[1:3,]
#>    total_wc_results result_number       oclc          isbn   issn
#>              <char>         <int>     <char>        <char> <char>
#> 1:              986             1 1125170419 9782253183464   <NA>
#> 2:              986             2 1049849403 9788415618843   <NA>
#> 3:              986             3 1203070641 9781664921993   <NA>
#>              title             author pub_date lang_code      bib_level
#>             <char>             <char>    <int>    <char>         <char>
#> 1: Madame Bovary : Flaubert, Gustave,     2019       fre Monograph/Item
#> 2: Madame Bovary / Flaubert, Gustave,     2019       spa Monograph/Item
#> 3: Madame Bovary / Flaubert, Gustave,     2021       eng Monograph/Item
#>                   record_type pub_place_code          publisher
#>                        <char>         <char>             <char>
#> 1:          Language Material             fr le Livre de poche,
#> 2:          Language Material             sp               <NA>
#> 3: Nonmusical sound recording            ohu               <NA>
#>                      leader                                     oh08
#>                      <char>                                   <char>
#> 1: 00000cam a2200000Mi 4500 190619s2019    fr a   g      000 1 fre d
#> 2: 00000cam a2200000Ii 4500 180827t20192018sp a          000 1 spa d
#> 3: 00000cim a2200000Mi 4500 201104s2021    ohunnnneq      f  n eng d
#>                                                    query
#>                                                   <char>
#> 1: srw.ti="Madame Bovary" and srw.au="Gustave Flauber...
#> 2: srw.ti="Madame Bovary" and srw.au="Gustave Flauber...
#> 3: srw.ti="Madame Bovary" and srw.au="Gustave Flauber...

This should give you an idea of the rich information returned by the results data.table. All of the information returned is from (or derived from) the MARCXml that the API returns, save for

• total_wc_results Which returns the number of the results the query yields in WorldCat even if the number of records/results requested is lower than this number.

• result_number This number of the result, which is helpful if you’re using a starting position other than 1 (the default)

• query The final SRU query (after any of the SRU query assistance routines step in) that is sent to the API. This can be useful for debugging.

Hereafter, the output of the example queries will be abbreviated, truncated, elided, or only show a subset of columns in order to save space and aid in following the guide.

What does an error look like

If you craft a query that yields no results, a message telling you such will be displayed, and the return value is NULL.

If you make an error in the query syntax, no results will be returned, but a diagnostic message returned from the server may tell you what went wrong. Here are two examples…

# missing ending double quotes
worldcat_api_search('$title="Madame Bovary and $langauge=greek')
#> Received diagnostic message: Query syntax error (org.z3950.zing.cql.CQLParseException: expected index or term, got EOF)
#> no results found
#> NULL

# "$titley" is not a valid search index
worldcat_api_search('$titley="Madame Bovary" and $langauge=greek')
#> Received diagnostic message: Unsupported index (srw.tiy)
#> no results found
#> NULL

worldcat_api_search("$holding_library=NYP")
#> Received diagnostic message: Limit index, can only be used to narrow a result
#> for a non-limit index (srw.li)
#> no results found
#> NULL

This last one failed because $holding_library is considered a limit index. This means that you can’t use it to search directly, you can only combine with search facet with other non-limiting indexes to filter the results. Hopefully this will make more sense as you go through this vignette.

worldcat_api_search by example / quick start

In guides like this, there is frequently tension between wanting to be a complete reference, and wanting to show cool/helpful examples at a glance.

In service of easing this tension, we’ll first look at some examples illustrating things this function can do and then turn our attention to an adequate explanation of boolean operators, relations operators, search indexes, etc…

These examples are largely taken from the examples section of the R documentation for this function.

These examples (and the ones further along in this vignette) will also use data.table syntax to limit number of rows and columns returned to aid reading.

We’ll be focusing on the types of queries you can use in this section; controlling the behavior of the function via changing parameters will come later.

library(libbib)

# title search for "The Brothers Karamazov"
results <- worldcat_api_search('$title="Brothers Karamazov"')

# Madame Bovary by Gustave Flaubert in Greek
sru <- '$author="Gustave Flaubert" and
          $title="Madame Bovary" and
          $language=greek'
results <- worldcat_api_search(sru)

# Hip Hop materials on wax, cassette, or CD published from
# 1987 to 1990
sru <- '(($material_type=cas or $material_type=cda or $material_type=lps)
           and $subject="Rap") and $year="1987-1990"'
results <- worldcat_api_search(sru)

# keyword search for "Common Lisp" for materials held
# at The New York Public Library
sru <- '$keyword="common lisp" and $holding_library=NYP'
results <- worldcat_api_search(sru)

# keyword search for "Common Lisp" for materials held
# by any of the members of the "Manhattan Research
# Library Initiative" (MaRLI) joint borrowing program
# (New York Public Library, Columbia University, and
# New York University)
sru <- '($keyword="common lisp" and $holding_library=NYP)
          or ($keyword="common lisp" and $holding_library=ZYU)
          or ($keyword="common lisp" and $holding_library=ZCU)'
results <- worldcat_api_search(sru)

# Books (only books) about Ethics (by dewey division 170s or
# LC call number subject class "BJ") published in the 19th
# century
sru <- '($dewey="17*" or $lc_call="bj*") and $year="18*" and
           $material_type=bks'
results <- worldcat_api_search(sru)

# Materials on Musicology (by Dewey division 780s) at
# the New York Public Library and not held by any
# other insitution
sru <- '$dewey="78*" and $holding_library=NYP and
          $library_holdings_group=11'
results <- worldcat_api_search(sru)

# Search for materials on "Danger Music" published since 2010
results <- worldcat_api_search('$keyword="danger music" and $year="2010-"')

Now that we’ve seen these examples, sans explanation, we can now have a closer look into the components of a query. Broadly speaking, there are four concepts to be aware of…

• relations operators

• boolean operators

• wildcards

• search indexes

We’ll be looking at each in this order, because I think that makes the most sense. To illustrate the first three concepts, though, we have to use search indexes, before a formal explanation of what they are.

Briefly, a search index is a facet along which to search. In all of the examples above, the search indexes were prefixed by a $ character (e.g. $title, author, $keyword, etc…)

Relations operators

There are four relations operators available for use…

• =

• exact

• any

• all

It should be noted that not every relations operator is available for use with every search index.

sru <- '$title+=+"Common Lisp"'
results <- worldcat_api_search(sru)
results[1:3, .(total_wc_results, title)]
#>    total_wc_results                   title
#>              <char>                  <char>
#> 1:              367           Common Lisp /
#> 2:              367 Practical Common Lisp /
#> 3:              367           Common LISP /

sru <- '$title = "Common Lisp"'
results <- worldcat_api_search(sru)
results[1:3, .(total_wc_results, title)]
#>    total_wc_results                   title
#>              <char>                  <char>
#> 1:              367           Common Lisp /
#> 2:              367 Practical Common Lisp /
#> 3:              367           Common LISP /

results <- worldcat_api_search('$title exact "Common Lisp"')
results[1:3, .(total_wc_results, title)]
#>    total_wc_results         title
#>              <char>        <char>
#> 1:               37 Common Lisp /
#> 2:               37 Common LISP /
#> 3:               37 Common LISP /

results <- worldcat_api_search('$title any "Common Lisp"')
results[1:3, .(total_wc_results, title)]
#>    total_wc_results                                                 title
#>              <char>                                                <char>
#> 1:           351777 An Inquiry into the Human Mind on the Principles o...
#> 2:           351777                            The book of common prayer.
#> 3:           351777                        Dictionary of Phrase and Fable

results <- worldcat_api_search('$title = Common or $title = "Lisp"')
results[1:3, .(total_wc_results, title)]
#>    total_wc_results                                                 title
#>              <char>                                                <char>
#> 1:           351777 An Inquiry into the Human Mind on the Principles o...
#> 2:           351777                            The book of common prayer.
#> 3:           351777                        Dictionary of Phrase and Fable

results <- worldcat_api_search('$title exact "Finnegans Wake"')
results[1:3, .(total_wc_results, title, query)]
#>    total_wc_results            title                         query
#>              <char>           <char>                        <char>
#> 1:              761 Finnegans wake / srw.ti exact "Finnegans Wake"
#> 2:              761   Finnegans Wake srw.ti exact "Finnegans Wake"
#> 3:              761   Finnegans wake srw.ti exact "Finnegans Wake"

# yields the same results as

results <- worldcat_api_search('$title exact "Finnegan\'s Wake"')
results[1:3, .(total_wc_results, title, query)]
#>    total_wc_results            title                         query
#>              <char>           <char>                        <char>
#> 1:              761 Finnegans wake / srw.ti exact "Finnegan's Wake"
#> 2:              761   Finnegans Wake srw.ti exact "Finnegan's Wake"
#> 3:              761   Finnegans wake srw.ti exact "Finnegan's Wake"

Boolean operators (and operator precedence)

As evinced in the examples shown earlier in this vignette, you can use the boolean operators and, or, and not to refine your search.

As you mix different boolean operators in a single query, care must be taken to ensure that the order/precedence of these operators matches your intention.

For example, in one of the examples above, we searched for Hip Hop materials on wax, cassette, or CD published from 1987 to 1990 with the following SRU query

'(($material_type=cas or $material_type=cda or $material_type=lps)
   and $subject="Rap") and $year="1987-1990"'

It’s important to note that if we wrote the query like shown below

$material_type=cas or $material_type=cda or $material_type=lps
   and $subject="Rap" and $year="1987-1990"'

without any parentheses, the meaning of the query would be ambiguous. Make sure you use parentheses around distinct sections of your search incantation to disambiguate the query.

Wildcards

taken verbatim from the archived documentation

For right truncation use an asterisk - * There must be at least three characters before the * for the query to work. There is no left truncation.

To wildcard a single character wildcard use a number sign - #. So a query for wom#n provides results that include both woman and women in the results.

For a 0-9 number of characters as wildcard characters use ?n. So a query for colo?1r provides results of color and colour.

To wildcard characters within (or at the end) a term use the question mark - ?. So a query for colo?r provides results of color, colour, colonizer, and colorimeter.

Table of all available search indexes

Selected Search Indexes in focus

Options in the worldcat_api_search function

Besides for, of course, the SRU query, the worldcat_api_search function takes a number of optional parameters that can be used to alter its semantics. Below is a list and explanation of each of those parameters. (This information is also available by running help("worldcat_api_search") in an R console after loading the libbib package.)

• max_records The maximum number of search results to return. This must be a number between 0 and 100 or Inf. If Inf, the function will automatically make all follow-up requests to retrieve all search results. To limit the number of times the API is hit, the default is 10 search results.

• sru_query_assist A logical (boolean) indicating whether translation from more human-readable aliases to the SRU search index codes should be allowed. The default is TRUE. It can also be set to FALSE. You can control this parameter globally by setting options("libbib.sru_query_assist").

• frbrGrouping FRBR (Functional Requirements for Bibliographic Records) is a conceptual framework for understanding the relationships between a “work” (e.g. a novel as planned by an author), its “expression” (e.g. the manuscript of that novel), it’s “manifestation” (e.g. the first published version of the novel), and, finally, an “item” (e.g an actual physical book [or microform; audio file; etc…] of the work). With the frbrGrouping parameter set to on (default), an attempt is made by the WorldCat API to group together similar editions and present only the top held record as the representative record for that group. This, conceptually, can be viewed as an attempt to return search results for any expression of the “work” (or works) referred to in the search query.

• start_at The search result to start at (default is 1)

• wskey A WorldCat API key. This function is easiest to use by setting the wskey globally, with the following incantation: options(libbib.wskey="YOUR WSKEY GOES HERE")

• more A logical indicating whether more information from the MARCXML search results should be returned (publisher, bib level, etc….). The default is TRUE.

• print.progress A logical indicating whether a message should be displayed for each API request. If max_records is Inf a message will be displayed for every group of 100 search results the function fetches. The default is TRUE.

• debug A logical indicating whether the HTTP and API responses should be printed (for debugging). The default is FALSE.

Combining this with the other API search functions in libbib

There are numerous ways to combine the worldcat_api_search with the other functions that libbib provides to do some really useful investigations. In the example below, will be using the worldcat_api_search and worldcat_api_locations_by_oclc functions to get a list of institutions that hold any edition of my textbook. (This example uses some data.table specific syntax for brevity, but it will work with base R [or “tidyverse”] translations just fine.)

First, let’s use the search function to get a list of all search results for the book…

results <- worldcat_api_search('srw.ti="Data Analysis with r"
                                  and srw.au=fischetti',
                                max_records=Inf)

# inspect some of the columns in the first 5 results
results[1:5, .(total_wc_results, result_number, oclc,
               title, author, pub_date)
#>    total_wc_results result_number       oclc                   title
#>              <char>         <int>     <char>                  <char>
#> 1:               11             1 1005106045 DATA ANALYSIS WITH R -.
#> 2:               11             2 1089176194  Data analysis with R :
#> 3:               11             3  949229431  Data analysis with R :
#> 4:               11             4 1242682069    Data Analysis with R
#> 5:               11             5 1242707288    Data Analysis with R
#>              author pub_date
#>              <char>    <int>
#> 1: FISCHETTI, TONY.     2018
#> 2: Fischetti, Tony.     2018
#> 3: Fischetti, Tony.     2015
#> 4:  Fischetti, Tony     2015
#> 5:  Fischetti, Tony     2015

Now let’s get all the unique OCLC numbers from all the search results.

all_the_oclcs <- results[, unique(oclc)]
all_the_oclcs
#>  [1] "1005106045" "1089176194" "949229431"  "1242682069" "1242707288"
#>  [6] "1244405806" "1104264768" "1242685020" "1242707684" "1244406814"
#> [11] "1104846312"

On to the worldcat_api_locations_by_oclc function! Since this function takes one OCLC number at a time, we need to use a looping-construct to run the function with all the OCLC numbers in all_the_oclcs. We’ll be using the pblapply function (from the great pbapply package) to do this because we get a useful progress bar with no extra effort.

holds <- pblapply(all_the_oclcs,
                  function(x){
                    worldcat_api_locations_by_oclc(x,
                      include.bib.info=FALSE)
                    })

Since the pblapply function returns a list of data.tables (on for each OCLC), we’ll use data.table’s rbindlist function to one data.table containing all the results.

all_holdings_dt <- rbindlist(holds)
all_holdings_dt[1:3]
#>          oclc institution_identifier            institution_name copies
#>        <char>                 <char>                      <char> <char>
#> 1: 1005106045                    FEM        The Ferguson Library      1
#> 2: 1005106045                    YDX        YBP Library Services      1
#> 3: 1005106045                    DUQ Duquesne University Library      1

all_holdings_dt[, .(institution_name)]
#>                                            institution_name
#>                                                      <char>
#>    1:                                  The Ferguson Library
#>    2:                                  YBP Library Services
#>    3:                           Duquesne University Library
#>    4:                                    Centennial College
#>    5:                                  George Brown College
#>   ---
#> 1029:        Hochschule Mittweida (FH), Hochschulbibliothek
#> 1030:                                           Cyberlibris
#> 1031:                                           Cyberlibris
#> 1032:                                           Cyberlibris
#> 1033: Bibliothèque de l'Université du Québec à Trois-Riv...

There you have it! My textbook is held by 1033 distinct OCLC institutions!

Although the example above only searches the holding institutions of one specific book, the idiom is most helpful/interesting/cool when used for finding the holding institution of a entire class of materials.

For example, in a recent project for a curator at my institution, I used the Search API to find all search results for materials on a very specific topic, got all the holding institutions for each of the search results, and then aggregated the institutions (with this package’s dt_counts_and_percents function) to find the institutions holding the most items on this particular (very specific) topic.

Sundry external / archived documentation links

• WorldCat Search API current documentation

  scroll to SRU

  Current documentation on the (SRU) WorldCat Search API call. Contains all index names and codes, parameters available, and the meaning of API status code return numbers

  https://developer.api.oclc.org/wcv1

• WorldCat Search API > Using the API > Request Types > SRU

  (archived OCLC documentation link from 2013)

  Examples of simple SRU requests and explanation of frbrGrouping and servicelevel parameters

  https://web.archive.org/web/20130915101249/http://oclc.org/developer/documentation/worldcat-search-api/sru

• WorldCat Search API > Indexes

  (archived OCLC documentation link from 2013)

  Explation of relations operators, boolean operators, title index stop-words (removed words), the MARC subfields search by (only) the subject index, and explation of the wildcard characters *; #; and ?.

  https://web.archive.org/web/20130530034745/http://oclc.org/developer/documentation/worldcat-search-api/indexes

• WorldCat Search API > Indexes > Complete List of Indexes

  (archived OCLC documentation link from 2013)

  Explation of relations operators and a complete list of all indexes, their respective SRU index code, the relations available for use with each, and the MARC fields search for each index. Here, the SRU index codes have a prefix of sru, but it’s really srw.

  https://web.archive.org/web/20130915101232/http://oclc.org/developer/documentation/worldcat-search-api/complete-list-indexes

• Material Types Names and Codes

  A complete list of Material Types you can search for, their codes

  For use with the srw.mt/$material_type index.

  Material Type names and codes - OCLC Support

  Archived from original for posterity

• WorldCat Search API > Indexes > Tips for specific indexes

  (archived OCLC documentation link from 2013)

  Contains a lot of very helpful information about some of the different search indexes, the stop words they use, normalization rules, and a very helpful cross-walk on the “Number of Holding Libraries” index.

  https://web.archive.org/web/20120311003221/http://oclc.org/developer/documentation/worldcat-search-api/tips-specific-indexes

• WorldCat Search API > Using the API > Parameters

  (archived OCLC documentation link from 2013)

  Helpful information about what the different SRU search parameters mean. Remember that the parameters of the worldcat_api_search function (a) only contain a subset of these, and (b) may have slightly different names https://web.archive.org/web/20121101034843/http://oclc.org/developer/documentation/worldcat-search-api/parameters

• Finding Institution Codes

  A search box for searching institution names and returning instituion codes suitable for use in the holding institution search index.

  https://www.oclc.org/en/contacts/libraries.html

• Searching WorldCat Indexes

  (archived OCLC documentation link from 2012)

  Information about search indexes that are largely available in the above links. Not related to the (SRU) Search API but could be helpful, anyway.

  https://web.archive.org/web/20120509072600/http://www.oclc.org/us/en/support/documentation/worldcat/searching/searchworldcatindexes/default.htm

• CQL specification

  Helpful information about CQL query syntax. Not all is applicable to the WorldCat Search API that is discussed here.

  https://www.loc.gov/standards/sru/cql/spec.html

• Searching WorldCat Indexes

  Current page with links containing a lot of information about wielding WorldCat searches. Not all information is application to the WorldCat Search API that is discussed here, and most of the most helpful information is already covered in the links above.

  https://help.oclc.org/Librarian_Toolbox/Searching_WorldCat_Indexes

Similar projects / code

• A Ruby Gem to communicate with the WorldCat Search API

  https://github.com/vivien/worldcat

• Python scripts using the WorldCat search API

  https://github.com/ysugiyama3/worldcat-search-api

Papers/Articles

• Using the WorldCat API to Develop Data-Driven Decision-Making for Gifts-in-Kind

  https://journals.ala.org/index.php/lrts/article/view/7074/9638

• Destroyer and Preserver, Hear, Oh Hear! Not All Uncirculated Books Must Chariotest to a Dark Wintry Bed: How We Used the Books Must Chariotest to a Dark Wintry Bed: How We Used the OCLC WorldCat Search API to Inform Our Weeding Decisions with OCLC WorldCat Search API to Inform Our Weeding Decisions with Holdings Data Holdings Data

  https://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=2026&context=charleston