# API and Database Reference

## Overview

metasurvey provides a REST API built with plumber backed by MongoDB for sharing recipes, workflows, and variable metadata with the community. The API can be self-hosted (see `vignette("self-hosting")`) and is used by both the R client functions (`api_*`) and the Shiny exploration application.

After deploying, the Swagger UI interface at `<your-api-url>/__docs__/` provides an interactive endpoint explorer automatically generated by plumber. For detailed request/response schemas and MongoDB collection documentation, see the sections below.

## Configuration

    library(metasurvey)

    # Point to your self-hosted API
    configure_api("https://your-api-host.example.com")

    # Or use an environment variable
    Sys.setenv(METASURVEY_API_URL = "https://your-api-host.example.com")

The R client reads the URL first from `configure_api()`, then falls back to the `METASURVEY_API_URL` environment variable.

## Authentication

The API uses JWT (JSON Web Token) authentication with HMAC-SHA256 signing. Tokens expire after 24 hours; long-lived tokens (90 days) can be generated for automated scripts.

### Registration

    # Individual account (auto-approved)
    api_register("Ana Garcia", "ana@example.com", "password123")

    # Institutional member (requires admin review)
    api_register(
      "Carlos Rodriguez",
      "carlos@ine.gub.uy",
      "password123",
      user_type = "institutional_member",
      institution = "INE Uruguay"
    )

Account types:

Type | Description | Approval
--- | --- | ---
`individual` | Independent researcher | Automatic
`institutional_member` | Member of a recognized institution | Requires admin review
`institution` | Institutional account | Requires admin review

### Login

    api_login("ana@example.com", "password123")

The token is stored in the session and used automatically in subsequent API calls. The client automatically renews tokens within 5 minutes of their expiration.

### Session Management

    # View current user profile
    api_me()

    # Refresh token
    api_refresh_token()

    # Logout
    api_logout()

### Long-lived Tokens

For automated scripts and CI/CD, generate a 90-day token from the Shiny application (Profile tab) or use it directly:

    Sys.setenv(METASURVEY_TOKEN = "your-long-lived-token")

    # API calls work without interactive login
    recipes <- api_list_recipes(survey_type = "ech")

## API Endpoints

### Recipes

Method | Endpoint | Auth | Description
--- | --- | --- | ---
`GET` | `/recipes` | No | List and search recipes
`GET` | `/recipes/:id` | No | Get an individual recipe
`POST` | `/recipes` | Yes | Publish a new recipe
`POST` | `/recipes/:id/download` | No | Increment download counter

#### List Recipes

    # All recipes
    all <- api_list_recipes()

    # Filter by survey type
    ech <- api_list_recipes(survey_type = "ech")

    # Search by text
    labor <- api_list_recipes(search = "empleo")

    # Filter by topic
    income <- api_list_recipes(topic = "income")

    # Filter by certification level
    official <- api_list_recipes(certification = "official")

    # Pagination
    page2 <- api_list_recipes(limit = 10, offset = 10)

Query parameters:

Parameter | Type | Description
--- | --- | ---
`search` | string | Regex search on recipe name
`survey_type` | string | `ech`, `eaii`, `eph`, `eai`
`topic` | string | `labor_market`, `income`, `education`, `health`, `demographics`, `housing`
`certification` | string | `community`, `reviewed`, `official`
`user` | string | Filter by author email
`limit` | integer | Maximum results (default 50)
`offset` | integer | Skip N results (default 0)

#### Get Recipe

    recipe <- api_get_recipe("ech_employment_001")

#### Publish Recipe

    api_login("ana@example.com", "password123")
    api_publish_recipe(my_recipe)

The server automatically sets the `user` field from the JWT, initializes `downloads = 0`, generates an `id` if not provided, and assigns the `community` certification by default.

### Workflows

Method | Endpoint | Auth | Description
--- | --- | --- | ---
`GET` | `/workflows` | No | List and search workflows
`GET` | `/workflows/:id` | No | Get an individual workflow
`POST` | `/workflows` | Yes | Publish a new workflow
`POST` | `/workflows/:id/download` | No | Increment download counter

    # List workflows for ECH
    wf <- api_list_workflows(survey_type = "ech")

    # Find workflows that use a specific recipe
    wf <- api_list_workflows(recipe_id = "ech_employment_001")

    # Get specific workflow
    w <- api_get_workflow("wf_labor_market_001")

    # Publish
    api_publish_workflow(my_workflow)

### ANDA Variable Metadata

> Note: The ANDA integration is an unofficial implementation that parses DDI XML metadata from INE Uruguay’s public ANDA catalog. It is not endorsed by INE and may contain errors or become outdated if INE changes the catalog structure. Always verify critical variable definitions against the official codebook.

The `/anda/variables` endpoint provides variable metadata obtained from INE Uruguay’s ANDA catalog (DDI XML format). This includes variable labels, value categories, and type information.

Method | Endpoint | Auth | Description
--- | --- | --- | ---
`GET` | `/anda/variables` | No | Get variable metadata

    # Get all ECH variables
    vars <- api_get_anda_variables(survey_type = "ech")

    # Get specific variables
    vars <- api_get_anda_variables(
      survey_type = "ech",
      var_names = c("pobpcoac", "e27", "ht11")
    )

Query parameters:

Parameter | Type | Description
--- | --- | ---
`survey_type` | string | Survey type (default `"ech"`)
`names` | string | Comma-separated variable names (all if empty)

Each variable document contains:

Field | Description
--- | ---
`name` | Variable name (lowercase)
`label` | Human-readable label
`type` | `discrete`, `continuous`, or `unknown`
`value_labels` | List of code-label mappings
`description` | Extended description
`source_edition` | Survey edition (e.g., `"2024"`)
`source_catalog_id` | ANDA catalog ID (e.g., `767`)

### Administration

Method | Endpoint | Auth | Description
--- | --- | --- | ---
`GET` | `/admin/pending-users` | Admin | List institutional accounts pending review
`POST` | `/admin/approve/:email` | Admin | Approve an institutional account
`POST` | `/admin/reject/:email` | Admin | Reject an institutional account

Admin access is controlled via the `METASURVEY_ADMIN_EMAIL` environment variable on the server.

### Health Check

Method | Endpoint | Auth | Description
--- | --- | --- | ---
`GET` | `/health` | No | API and MongoDB status

    {
      "status": "ok",
      "service": "metasurvey-api",
      "version": "2.0.0",
      "database": "metasurvey",
      "mongodb": "connected",
      "timestamp": "2026-02-15T12:00:00Z"
    }

## MongoDB Schema

The database has four collections, each with JSON Schema validation and optimized indexes.

### Entity-Relationship Diagram

The following diagram shows the MongoDB collections and their relationships:

      ┌──────────────────┐       ┌──────────────────────┐
      │     users         │       │      recipes          │
      ├──────────────────┤       ├──────────────────────┤
      │ email (PK)       │──┐    │ id (PK)              │
      │ name             │  │    │ name                 │
      │ password_hash    │  ├───>│ user (FK)            │
      │ user_type        │  │    │ survey_type          │
      │ institution      │  │    │ edition              │
      └──────────────────┘  │    │ steps[]              │
                            │    │ certification{}      │
                            │    │ categories[]         │
                            │    └──────────┬───────────┘
                            │               │
                            │    ┌──────────┴───────────┐
                            │    │     workflows         │
                            │    ├──────────────────────┤
                            │    │ id (PK)              │
                            └───>│ user (FK)            │
                                 │ survey_type          │
                                 │ recipe_ids[] (FK)    │
                                 │ calls[]              │
                                 └──────────────────────┘

      ┌──────────────────────┐
      │   anda_variables      │
      ├──────────────────────┤
      │ survey_type (PK)     │
      │ name (PK)            │
      │ label                │
      │ type                 │
      │ value_labels{}       │
      └──────────────────────┘

      Relationships:
        users    ──1:N──>  recipes     (publishes)
        users    ──1:N──>  workflows   (publishes)
        recipes  ──1:N──>  workflows   (referenced by)

### Collections

#### `users`

Field | Type | Required | Description
--- | --- | --- | ---
`name` | string | Yes | Display name
`email` | string | Yes | Email (unique, validated)
`password_hash` | string | Yes | SHA-256 hash (64 characters)
`user_type` | enum | Yes | `individual`, `institutional_member`, `institution`
`institution` | string | No | Institution name
`verified` | boolean | No | Whether identity is verified
`review_status` | enum | No | `approved`, `pending`, `rejected`
`reviewed_by` | string | No | Reviewing admin’s email
`reviewed_at` | string | No | ISO timestamp
`created_at` | string | Yes | ISO timestamp

Indexes: unique on `email`.

#### `recipes`

Field | Type | Required | Description
--- | --- | --- | ---
`id` | string | No | Unique identifier (auto-generated)
`name` | string | Yes | Recipe name
`user` | string | Yes | Author email
`survey_type` | enum | Yes | `ech`, `eaii`, `eph`, `eai`
`edition` | string/array | No | Survey edition(s)
`description` | string | No | Description
`topic` | enum | No | `labor_market`, `income`, `education`, `health`, `demographics`, `housing`
`version` | string | No | Semantic version (default `"1.0.0"`)
`downloads` | number | No | Download counter (default `0`)
`steps` | array | No | Step expressions as strings
`depends_on` | array | No | Required input variable names
`depends_on_recipes` | array | No | IDs of dependent recipes
`categories` | array | No | Category objects
`certification` | object | No | `{level, certified_at, certified_by, notes}`
`user_info` | object | No | `{name, user_type, email, url, verified}`
`doc` | object | No | `{input_variables, output_variables, pipeline}`
`data_source` | object | No | `{s3_bucket, s3_prefix, file_pattern, provider}`

Indexes: unique on `id`; on `user`, `survey_type`, `topic`, `downloads` (desc), `certification.level`; compound on `(survey_type, edition)`; text search on `(name, description, topic)`.

#### `workflows`

Field | Type | Required | Description
--- | --- | --- | ---
`id` | string | No | Unique identifier (auto-generated)
`name` | string | Yes | Workflow name
`user` | string | Yes | Author email
`survey_type` | enum | Yes | `ech`, `eaii`, `eph`, `eai`
`edition` | string/array | No | Survey edition(s)
`description` | string | No | Description
`version` | string | No | Semantic version
`downloads` | number | No | Download counter
`estimation_type` | string/array | No | `annual`, `quarterly`, `monthly`
`recipe_ids` | array | No | Referenced recipe IDs
`calls` | array | No | Estimation calls as strings
`call_metadata` | array | No | Call descriptions
`categories` | array | No | Category objects
`certification` | object | No | Same as recipes
`user_info` | object | No | Same as recipes

Indexes: unique on `id`; on `user`, `survey_type`, `recipe_ids`, `downloads` (desc); compound on `(survey_type, edition)`; text search on `(name, description)`.

#### `anda_variables`

Field | Type | Required | Description
--- | --- | --- | ---
`survey_type` | string | Yes | Survey type
`name` | string | Yes | Variable name (lowercase)
`label` | string | Yes | Human-readable label
`type` | enum | No | `discrete`, `continuous`, `unknown`
`value_labels` | object | No | Code-label mappings
`description` | string | No | Extended description
`source_edition` | string | No | Edition (e.g., `"2024"`)
`source_catalog_id` | number | No | ANDA catalog ID

Indexes: compound unique on `(survey_type, name)`; on `survey_type`.

## Database Setup

To set up the database on a new deployment:

    # 1. Create collections with JSON Schema validation and indexes
    mongosh "$METASURVEY_MONGO_URI" inst/scripts/setup_mongodb.js

    # 2. Seed recipes, workflows, and users
    METASURVEY_MONGO_URI="..." Rscript inst/scripts/seed_ech_recipes.R

    # 3. Seed ANDA variable metadata from INE catalog
    METASURVEY_MONGO_URI="..." Rscript inst/scripts/seed_anda_metadata.R

The setup script creates the four collections and builds the indexes. It is idempotent: existing collections are skipped.

## Server Deployment

### Environment Variables

Variable | Required | Default | Description
--- | --- | --- | ---
`METASURVEY_MONGO_URI` | Yes | — | MongoDB connection string
`METASURVEY_DB` | No | `metasurvey` | Database name
`METASURVEY_JWT_SECRET` | No | `metasurvey-dev-secret-...` | JWT signing secret (override in production)
`METASURVEY_ADMIN_EMAIL` | No | — | Admin email for institutional review

### Running Locally

    METASURVEY_MONGO_URI="mongodb+srv://user:pass@cluster.mongodb.net" \
      Rscript -e 'plumber::plumb("inst/api/plumber.R")$run(port = 8787)'

The Swagger UI interface will be available at `http://localhost:8787/__docs__/`.

### Docker

    docker build -t metasurvey-api inst/api/
    docker run -p 8787:8787 \
      -e METASURVEY_MONGO_URI="mongodb+srv://..." \
      -e METASURVEY_JWT_SECRET="your-production-secret" \
      -e METASURVEY_ADMIN_EMAIL="admin@example.com" \
      metasurvey-api

### Railway

The API is configured for Railway deployment via the `render.yaml` file in `inst/api/`. Push the repository and configure the environment variables in the Railway dashboard.

## CORS

The API allows cross-origin requests from any origin:

- Allowed methods: GET, POST, OPTIONS
- Allowed headers: Content-Type, Authorization

## Next Steps

- Interactive recipe explorer – Browse recipes and workflows through the Shiny web application
- Creating and publishing recipes – Build recipes programmatically and publish them to the API
- Estimation workflows – Compute weighted survey estimates with `workflow()`