Getting started with datapond

The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

datapond is a registry of curated DuckDB databases built from public data. Each database is a single .duckdb file hosted on Hugging Face. This package attaches those files over HTTP so that only the byte ranges your query touches are transferred, or downloads them once for local use.

Browse

library(datapond)
dp_list()
dp_databases()[, c("id", "rows", "size_gb", "data_date_range")]
dp_info("eoir")

Query remotely

dp_connect() returns an ordinary DuckDB DBIConnection:

con <- dp_connect("eoir")
DBI::dbGetQuery(con, "SELECT * FROM proceedings LIMIT 5")

Every database ships a data dictionary; dp_describe() reads it:

dp_describe(con)                         # tables
dp_describe(con, table = "proceedings")  # columns, types, null %, join hints
dp_describe(con, search = "judge")       # columns by name

dbplyr works unchanged:

library(dplyr)
tbl(con, "proceedings") |>
  count(DEC_CODE, sort = TRUE) |>
  head(10) |>
  collect()
dp_disconnect(con)

Several databases

Pass a vector of ids. Tables are then qualified by database id, and ids that contain a hyphen must be double-quoted in SQL:

con <- dp_connect(c("cms-medicare", "openpayments"))
DBI::dbGetQuery(con, 'SELECT * FROM "cms-medicare".physician_summary LIMIT 5')
tbl(con, I('"openpayments".main.general_payments')) |> head(5) |> collect()
dp_disconnect(con)

Local copies

Remote mode is cheap for counts and filters but SELECT * over a large table transfers most of the file. Download once for disk speed:

dp_download("dol-visas")
con <- dp_connect("dol-visas", local = TRUE)
dp_update("dol-visas")   # re-download only if the registry is newer

Set options(datapond.data_dir = "~/.datapond") to share downloads with the Python package.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.