---
title: "Modern surveillance data from OpenDataSUS"
author: "Renato Prado Siqueira"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Modern surveillance data from OpenDataSUS}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, collapse = TRUE, comment = "#>")
library(datasus)
```

## Overview

OpenDataSUS publishes record-level surveillance files separately from
TABNET. `datasus` provides a generic catalog client and convenience functions
for frequently used datasets:

| Dataset | Function | Standardization key |
|:--|:--|:--|
| Serious adverse events following immunization | `esavi()` | `"esavi"` |
| Influenza-like illness notifications | `esus_sindrome_gripal()` | `"sindrome_gripal"` |
| Individual PNI doses | `pni_doses()` | `"pni_doses"` |
| COVID-19 hospital occupancy | `ocupacao_hospitalar()` | `"ocupacao_hospitalar"` |

## Search before downloading

Search the catalog and inspect its resources before requesting a large file:

```{r discovery, eval=FALSE}
opendatasus_catalogo("ESAVI")
opendatasus_catalogo("doses aplicadas PNI")

resources <- opendatasus_recursos("esavi")
resources[, c("id", "nome", "formato", "ano", "tamanho")]
```

`"last"` follows the latest partition found in the live catalog. Use an
explicit year and month when the analysis must remain reproducible.

## Start with selected columns

The convenience functions accept `n_max` for exploratory reads and `colunas`
to avoid parsing fields that are not needed:

```{r wrappers, eval=FALSE}
events <- esavi(
  n_max = 1000,
  colunas = c(
    "nu_notificacao", "dt_notificacao", "nu_idade", "ds_sexo"
  ),
  normalizar = TRUE
)

illness <- esus_sindrome_gripal(
  uf = "MS",
  ano = 2024,
  n_max = 1000,
  colunas = c(
    "dataNotificacao", "municipioIBGE", "idade", "sexo"
  ),
  normalizar = TRUE
)

doses <- pni_doses(
  ano = 2026,
  mes = 1,
  n_max = 1000,
  colunas = c(
    "co_paciente", "dt_vacina", "co_vacina",
    "co_municipio_paciente"
  ),
  normalizar = TRUE
)

occupancy <- ocupacao_hospitalar(
  ano = 2022,
  n_max = 1000,
  colunas = c(
    "dataNotificacao", "cnes", "ocupacaoHospitalarUti"
  ),
  normalizar = TRUE
)
```

Column names supplied to `colunas` are the names in the source file.
With `normalizar = TRUE`, the returned names are the stable analysis names
defined by the package dictionary.

## Standardize an existing data frame

Standardization can also be applied after data have been imported elsewhere.
This offline example uses fields from the ESAVI dictionary:

```{r standardize}
raw_events <- data.frame(
  nu_notificacao = c("A-001", "A-002"),
  dt_notificacao = c("2026-01-10", "2026-01-11"),
  nu_idade = c("34", "67"),
  ds_sexo = c("Feminino", "Masculino"),
  stringsAsFactors = FALSE
)

events <- datasus_padronizar(raw_events, sistema = "esavi")
str(events)
```

The dictionary documents source names, standardized names, semantic labels
and expected classes:

```{r dictionary}
head(datasus_dicionario("esavi"), 8)
```

## Detect schema drift

`datasus_validar_esquema()` checks whether important fields are present and
whether their classes agree with the curated schema:

```{r schema}
validation <- datasus_validar_esquema(
  events,
  sistema = "esavi",
  campos = c(
    "id_notificacao", "data_notificacao", "idade", "sexo"
  )
)
validation
```

Set `estrito = TRUE` in automated pipelines to stop when a required field is
missing or has an incompatible class:

```{r strict-schema, eval=FALSE}
datasus_validar_esquema(
  events,
  sistema = "esavi",
  campos = c("id_notificacao", "data_notificacao"),
  estrito = TRUE
)
```

## Resources split into multiple physical files

Some historical influenza-like illness resources publish their physical
files as links in the resource description. Expand them before building a
download plan:

```{r multipart, eval=FALSE}
resources <- opendatasus_recursos(
  "notificacoes-de-sindrome-gripal-leve-2020"
)
ms_id <- resources$id[
  resources$formato == "CSV" & grepl("^Dados MS", resources$nome)
][1]

files <- opendatasus_arquivos(
  "notificacoes-de-sindrome-gripal-leve-2020",
  recurso = ms_id,
  formato = "CSV"
)
files[, c("recurso", "ano", "parte", "url")]
```

`esus_sindrome_gripal()` reads all these parts transparently and applies
`n_max` across the combined result, rather than independently to every file.

## Provenance

Downloaded data retain the resource identifier, official URLs, update and
download times, local cache paths and checksums:

```{r provenance, eval=FALSE}
provenance <- datasus_proveniencia(events)
str(provenance)
```

Use `atualizar = TRUE` to ignore a cached copy and obtain the current portal
version.