Introduction to datacaged

The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

Overview

The datacaged package simplifies access to CAGED microdata (Cadastro Geral de Empregados e Desempregados) directly from HuggingFace, loading data into a local DuckDB database for efficient analysis.

It supports three series:

Period	Series	Database table
Jan/2020 – present	Novo CAGED	`caged_mov`, `caged_for`, `caged_exc`
Jan/1992 – Dec/2019	Legacy CAGED	`caged_antigo`
Jan/1992 – Dec/2019	CAGED Adjustments	`caged_ajustes`

Installation

# Via remotes
remotes::install_github("gecomt/datacaged")

Platform compatibility

The datacaged package is compatible with Windows, macOS and Linux with no extra configuration for Novo CAGED (2020+).

Recurso	Windows	macOS	Linux
Download (HTTPS)	OK	OK	OK
Downloads paralelos	OK	OK	OK
Novo CAGED (2020+)	OK	OK	OK
Legacy CAGED (pre-2020, PPMd)	(!) requires 7-Zip	(!) requires 7-Zip	(!) requires 7-Zip

Legacy CAGED uses PPMd compression. If you need this data, install 7-Zip:

Windows: baixe o instalador e instale normalmente
macOS: brew install 7-zip
Linux: sudo apt install 7zip ou sudo dnf install 7zip

Novo CAGED (2020+) uses LZMA, natively supported by the archive package on all platforms.

Parallel downloads

By default, the package downloads 3 files simultaneously (MOV, FOR and EXC for each month), resulting in approximately 3× faster downloads compared to sequential mode.

# Control the number of workers
caged_download(years = 2023, months = 1:3, workers = 3)  # padrão

# Set globally for the entire session
options(datacaged.workers = 4)

# Sequential mode (useful for unstable connections)
caged_download(years = 2023, months = 1, workers = 1)

Basic usage: Full pipeline

The caged_load() function does everything in a single command: downloads .7z files from HuggingFace, extracts, normalises and writes to DuckDB.

library(datacaged)

# Download Novo CAGED Jan–Dec/2023
# Novo CAGED: national file, `states` does not filter
caged_load(
  years    = 2023,
  months   = seq_len(12L),
  db_path = "caged.duckdb"
)

Progress is displayed in the terminal with a progress bar and final summary.

Querying the data

After populating the database, connect and query with dplyr or plain SQL:

library(dplyr)

con <- caged_connect("caged.duckdb")

# Monthly employment balance in 2023
saldo_mensal <- tbl(con, "caged_mov") |>
  group_by(competenciamov) |>
  summarise(saldo = sum(saldomovimentacao, na.rm = TRUE)) |>
  arrange(competenciamov) |>
  collect()

saldo_mensal

# Or with direct SQL
DBI::dbGetQuery(con, "
  SELECT
    competenciamov,
    uf,
    SUM(saldomovimentacao) AS saldo,
    AVG(salario)           AS salario_medio,
    COUNT(*)               AS movimentacoes
  FROM caged_mov
  WHERE uf = 35          -- Sao Paulo
  GROUP BY competenciamov, uf
  ORDER BY competenciamov
")

Always close the connection when done:

DBI::dbDisconnect(con, shutdown = TRUE)

Granular functions

For more control, use the functions individually:

1. Download files only

# Download and save to local cache (~/.local/share/R/datacaged por padrão)
manifest <- caged_download(
  years    = 2023,
  months   = c(1L, 2L, 3L),
  destdir = "~/meus_dados/caged_cache"
)

# manifest is a data.frame with the status of each file
dplyr::count(manifest, status)

2. Parse files manually

# One file at a time
df <- caged_parse("~/meus_dados/caged_cache/caged_mov/2023/CAGEDMOV202301.7z")
glimpse(df)

# Several at once
arquivos <- list.files(
  "~/meus_dados/caged_cache/NOVO_CAGED/2023",
  pattern    = "CAGEDMOV",
  full.names = TRUE
)
df_todos <- caged_parse_batch(arquivos)

3. Write to database

caged_to_duckdb(df_todos, db_path = "caged.duckdb")

Inspect the database

caged_info("caged.duckdb")
#> ── caged.duckdb ────────────────────────────────────────
#> Tamanho do arquivo: 142.3 MB
#> ── Tabelas ──────────────────────────────────────────────
#> * "caged_mov"   Registros: 3,665,155
#> * "caged_for"      Registros:    91,098
#> * "caged_exc"       Registros:     7,900
#>   Registros   : 4.823.901
#>   Competências: 202301 – 202312

Example: Historical series with legacy CAGED

# Baixa Legacy CAGED para Nordeste (2015–2019)
nordeste <- c("MA", "PI", "CE", "RN", "PB", "PE", "AL", "SE", "BA")

caged_load(
  years    = 2015:2019,
  db_path = "caged_historico.duckdb"
)

con <- caged_connect("caged_historico.duckdb")

# Evolução anual do saldo formal no Nordeste
tbl(con, "caged_antigo") |>
  mutate(ano = as.integer(substr(as.character(competencia), 1, 4))) |>
  group_by(ano, uf) |>
  summarise(saldo = sum(saldomovimentacao, na.rm = TRUE)) |>
  collect() |>
  tidyr::pivot_wider(names_from = uf, values_from = saldo)

DBI::dbDisconnect(con, shutdown = TRUE)

CAGED Adjustments

CAGED Adjustments contain retroactive corrections to legacy CAGED records (up to 2019). Use caged_adjustments_load() to download and write to the caged_ajustes table.

# Baixar ajustes de 2019
caged_adjustments_load(years = 2019, months = seq_len(12L), db_path = "caged.duckdb")

# Listar o que está disponível no HuggingFace
caged_hf_files(type = "ajustes")

# Comparar saldo original vs ajustado
con <- caged_connect("caged.duckdb")

antigo  <- dplyr::tbl(con, "caged_antigo")  |>
  dplyr::group_by(competencia) |>
  dplyr::summarise(saldo_original = sum(saldomovimentacao, na.rm = TRUE))

ajustes <- dplyr::tbl(con, "caged_ajustes") |>
  dplyr::group_by(competencia) |>
  dplyr::summarise(saldo_ajuste = sum(saldomovimentacao, na.rm = TRUE))

dplyr::full_join(antigo, ajustes, by = "competencia") |>
  dplyr::mutate(saldo_final = saldo_original + saldo_ajuste) |>
  dplyr::collect()

DBI::dbDisconnect(con, shutdown = TRUE)

Performance tips

DuckDB is columnar: prefer select() before collect() to bring only the necessary columns.
Re-runs are safe: caged_load() skips competencies already in the database by default. Use overwrite_competencies = TRUE to overwrite.
.7z cache: downloaded files are stored in ~/.local/share/R/datacaged. They can be reused without a new HuggingFace connection.
Memory: the pipeline processes one competency at a time to avoid excessive RAM usage with large periods.

Utilities

# Verificar se o HuggingFace está online antes de baixar
caged_status()

# Listar competências disponíveis no HuggingFace
caged_hf_files()                     # Novo CAGED (últimos 12 meses)
caged_hf_files(type = "antigo")      # Legacy CAGED
caged_hf_files(type = "ajustes")     # CAGED Adjustments

# Atualização incremental — baixa apenas o que ainda não está no banco
caged_update(db_path = "caged.duckdb")
caged_update(db_path = "caged.duckdb", series = c("novo", "antigo"))

# Exportar tabelas para Parquet (nativo DuckDB, muito rápido)
caged_to_parquet("caged.duckdb", output_dir = "~/exports")
caged_to_parquet("caged.duckdb", output_dir = "~/exports",
                 tables = "caged_mov", partition_by = "uf")

Main variables

Column	Description
`competenciamov`	Competency in Novo CAGED, `YYYYMM` format (ex: `202301`)
`competencia`	Competency in legacy CAGED and Adjustments, format `AAAAMM`
`uf`	IBGE state code (ex: `35` = SP)
`municipio`	IBGE municipality code
`saldomovimentacao`	`+1` hire, `-1` dismissal
`salario`	Contracted wage in BRL
`sexo`	`1` male, `3` female
`idade`	Age in years
`escolaridade`	Education level code (1–9)
`racacor`	Race/colour code (1–5)
`tipomovimentacao`	Reason for movement code
`secao`	CNAE 2.0 section (Novo CAGED)
`fonte_tipo`	`MOV`, `FOR`, `EXC`, `ANTIGO` or `AJUSTES`

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.