Metadata-driven programming

Lesson 5 — Clinical Programming with R

Lesson 5 of 12 Advanced ~90 min

Learning objectives

  • Explain why specifications should be machine-readable
  • Load and validate a specification with metacore
  • Apply metadata to a dataset with metatools
  • Drive derivations and checks from the specification
  • Keep the specification and the code in sync

The idea

A traditional specification is a Word document a human reads and then translates into code. Two artefacts, maintained separately, that drift.

Metadata-driven programming treats the specification as data: the code reads it, applies it, and checks against it. One artefact. If the specification says AGE is numeric with label “Age” and length 8, the code does not restate that — it applies it.

Traditional Metadata-driven
Spec in Word, code in R Spec is data the code reads
Labels typed into the program Labels applied from the spec
Define.xml written by hand at the end Define.xml generated from the spec
Spec change requires code change Spec change requires only a spec change
Drift discovered in QC Drift is impossible

The payoff scales: for one dataset it is barely worth it, for forty across five studies it is transformative.

metacore

metacore reads a specification into a standard R object with six related tables.

library(metacore)

meta <- spec_to_metacore("metadata/adam_spec.xlsx")

meta
#> Metacore object contains 6 datasets
#>   ds_spec     4 rows   (dataset level)
#>   ds_vars    112 rows  (dataset-variable link)
#>   var_spec    78 rows  (variable level)
#>   value_spec 145 rows  (value level)
#>   derivations 62 rows
#>   codelist    18 rows

The six tables:

Table Contains
ds_spec Dataset name, label, structure, key
ds_vars Which variables are in which dataset, order, core, key sequence
var_spec Variable name, label, type, length, format
value_spec Value-level: type/codelist/derivation per PARAMCD
derivations Derivation text, referenced by ID
codelist Controlled terminology, permitted values

Extract one dataset’s specification:

adsl_spec <- select_dataset(meta, "ADSL")

adsl_spec$var_spec |> head()
#> # A tibble: 6 x 6
#>   variable label                     type      length format common
#>   <chr>    <chr>                     <chr>      <int> <chr>  <lgl>
#> 1 STUDYID  Study Identifier          text          20 NA     TRUE
#> 2 USUBJID  Unique Subject Identifier text          30 NA     TRUE
#> 3 AGE      Age                       integer        8 NA     TRUE
#> 4 SEX      Sex                       text           1 NA     TRUE

adsl_spec$codelist
adsl_spec$derivations
TipSpecification format

spec_to_metacore() expects the CDISC-style Excel layout with sheets named Datasets, Variables, ValueLevel, Codelists, Dictionaries and Methods. The metacore package includes a template:

system.file("extdata", "ADaM_spec.xlsx", package = "metacore")

If your organisation’s spec has a different layout, write a converter once rather than restructuring every study’s spec.

metatools

Applies the metadata to a dataset.

library(metatools)

adsl <- adsl_raw |>
  drop_unspec_vars(adsl_spec) |>     # remove variables not in the spec
  check_variables(adsl_spec) |>      # error if a required variable is absent
  check_ct_data(adsl_spec) |>        # error on values outside the codelist
  order_cols(adsl_spec) |>           # apply the specified column order
  sort_by_key(adsl_spec)             # apply the specified sort

Each function does one thing and errors informatively:

check_variables(adsl, adsl_spec)
#> Error: Variables missing from the dataset: AGEGR1N, RACEN
#> Variables in the dataset but not the spec: TEMP_FLAG

That second line is as valuable as the first — a stray working variable that made it into the output is a common finding.

Building variables from the spec

# Create numeric companions from a codelist
adsl <- adsl |>
  create_var_from_codelist(
    metacore = adsl_spec,
    input_var = TRT01P,
    out_var   = TRT01PN
  )

# Build a categorical from a numeric using the spec's ranges
adsl <- adsl |>
  create_cat_var(
    metacore  = adsl_spec,
    ref_var   = AGE,
    grp_var   = AGEGR1,
    num_grp_var = AGEGR1N
  )

# Add SUPPQUAL variables to a parent domain
adae <- combine_supp(ae, suppae)

The full pattern

#-------------------------------------------------------------------------------
# Program: ad_adsl.R
# Purpose: Create ADSL, driven by metadata/adam_spec.xlsx
#-------------------------------------------------------------------------------

library(metacore); library(metatools); library(admiral)
library(xportr); library(dplyr); library(here)

# --- 1. Load the specification ---------------------------------------------
meta <- spec_to_metacore(here("metadata", "adam_spec.xlsx"))
spec <- select_dataset(meta, "ADSL")

# --- 2. Read source ---------------------------------------------------------
dm <- read_sdtm("DM"); ex <- read_sdtm("EX"); ds <- read_sdtm("DS")

# --- 3. Derive (admiral) ----------------------------------------------------
adsl <- dm |>
  mutate(TRT01P = ARM, TRT01A = ACTARM) |>
  derive_vars_merged(...) |>
  derive_var_trtdurd() |>
  mutate(SAFFL = if_else(!is.na(TRTSDT), "Y", "N"))

# --- 4. Apply the specification --------------------------------------------
adsl <- adsl |>
  create_var_from_codelist(spec, TRT01P, TRT01PN) |>
  create_cat_var(spec, ref_var = AGE, grp_var = AGEGR1, num_grp_var = AGEGR1N) |>
  drop_unspec_vars(spec) |>
  check_variables(spec) |>
  check_ct_data(spec, na_acceptable = TRUE) |>
  order_cols(spec) |>
  sort_by_key(spec)

# --- 5. Apply transport metadata and write ---------------------------------
adsl |>
  xportr_type(spec)     |>
  xportr_length(spec)   |>
  xportr_label(spec)    |>
  xportr_format(spec)   |>
  xportr_df_label(spec) |>
  xportr_write(here("data", "submission", "adsl.xpt"), strict_checks = TRUE)

saveRDS(adsl, here("data", "adam", "adsl.rds"))

The structure is always the same: derive, then conform, then write. Steps 4 and 5 are near-identical across every dataset in the study, which is exactly what you want — the only study-specific code is step 3.

Driving derivations from metadata

Beyond labels and types, the specification can drive the derivations themselves.

Parameter lookups

Instead of hard-coding a PARAMCD lookup in every BDS program:

# From the value-level metadata
param_lookup <- spec$value_spec |>
  filter(variable == "PARAMCD") |>
  select(PARAMCD = value, PARAM = where, origin, derivation_id)

adlb <- lb |>
  left_join(param_lookup, by = c("LBTESTCD" = "PARAMCD"))

Specification-driven categorisation

# metadata/categories.csv
# variable, source, lower, upper, label,   order
# AGEGR1,   AGE,    NA,    18,    "<18",   1
# AGEGR1,   AGE,    18,    65,    "18-64", 2
# AGEGR1,   AGE,    65,    NA,    ">=65",  3

apply_categories <- function(data, cat_spec) {
  vars <- unique(cat_spec$variable)

  for (v in vars) {
    rules <- filter(cat_spec, variable == v) |> arrange(order)
    src   <- unique(rules$source)

    conds <- purrr::pmap(rules, function(lower, upper, label, ...) {
      lo <- if (is.na(lower)) TRUE else rlang::expr(.data[[!!src]] >= !!lower)
      hi <- if (is.na(upper)) TRUE else rlang::expr(.data[[!!src]] <  !!upper)
      rlang::expr(!!lo & !!hi ~ !!label)
    })

    data <- mutate(data, "{v}" := case_when(!!!conds, .default = NA_character_))
  }
  data
}

adsl <- apply_categories(adsl, read_csv("metadata/categories.csv"))

Adding an age group now means editing a CSV, not a program.

Generated validation checks

# metadata/checks.csv
# dataset, check_id, description,                  expression
# ADSL,    ADSL001,  "USUBJID unique",             "!anyDuplicated(USUBJID)"
# ADSL,    ADSL002,  "SAFFL is Y or N",            "all(SAFFL %in% c('Y','N'))"
# ADSL,    ADSL003,  "TRTEDT >= TRTSDT",           "all(is.na(TRTEDT) | TRTEDT >= TRTSDT)"
# ADAE,    ADAE001,  "TRTEMFL requires TRTSDT",    "all(TRTEMFL != 'Y' | !is.na(TRTSDT))"

run_checks <- function(data, checks, dataset) {
  checks |>
    filter(dataset == !!dataset) |>
    rowwise() |>
    mutate(
      result = tryCatch(
        isTRUE(eval(parse(text = expression), envir = data)),
        error = function(e) NA
      ),
      status = case_when(isTRUE(result) ~ "PASS",
                         is.na(result)  ~ "ERROR",
                         .default       = "FAIL")
    ) |>
    ungroup() |>
    select(check_id, description, status)
}

run_checks(adsl, read_csv("metadata/checks.csv"), "ADSL")
#> # A tibble: 3 x 3
#>   check_id description         status
#>   <chr>    <chr>               <chr>
#> 1 ADSL001  USUBJID unique      PASS
#> 2 ADSL002  SAFFL is Y or N     PASS
#> 3 ADSL003  TRTEDT >= TRTSDT    FAIL
Warningeval(parse(text = ...)) deserves care

Evaluating strings from a file is powerful and dangerous. It is acceptable here because the checks file is a controlled study artefact under version control, reviewed like any other. It would not be acceptable if the expressions came from user input or an uncontrolled source.

A safer alternative for a shared framework is a small vocabulary of check types — unique, in_codelist, not_missing, range — with parameters, rather than arbitrary R expressions.

Keeping the spec and code in sync

The main failure mode of metadata-driven programming is a specification that has drifted from reality. Guard against it:

# 1. Every dataset must pass its own spec check
walk(names(datasets), function(nm) {
  check_variables(datasets[[nm]], select_dataset(meta, nm))
})

# 2. Every spec-referenced derivation must exist in code
spec_derivations <- meta$derivations$derivation_id
code_derivations <- grep("derivation_id", readLines("programs/adam/ad_adsl.R"), value = TRUE)

# 3. Compare the produced dataset's metadata against the spec
compare_to_spec <- function(data, spec) {
  actual <- tibble::tibble(
    variable = names(data),
    label    = purrr::map_chr(data, ~ attr(.x, "label") %||% NA_character_),
    type     = purrr::map_chr(data, ~ class(.x)[1]),
    max_len  = purrr::map_int(data, ~ if (is.character(.x))
                                max(c(0L, nchar(.x, type = "bytes")), na.rm = TRUE)
                              else NA_integer_)
  )

  spec$var_spec |>
    full_join(actual, by = "variable", suffix = c("_spec", "_actual")) |>
    mutate(
      missing_from_data = is.na(type_actual),
      missing_from_spec = is.na(type_spec),
      label_mismatch    = !is.na(label_spec) & !is.na(label_actual) &
                          label_spec != label_actual,
      length_exceeded   = !is.na(max_len) & !is.na(length) & max_len > length
    ) |>
    filter(missing_from_data | missing_from_spec | label_mismatch | length_exceeded)
}

compare_to_spec(adsl, spec)
#> # A tibble: 1 x 12
#>   variable label_spec  ... length_exceeded
#>   <chr>    <chr>           <lgl>
#> 1 DCSREAS  Reason for ...  TRUE          <- spec says 40, data has 47

Run this at the end of every ADaM program and fail the build on any row.

When metadata-driven is not worth it

Honestly:

  • A single small study — the setup cost exceeds the benefit
  • A spec that changes every day — you will spend more time on the spec format than on the analysis
  • Highly bespoke derivations — expressing them as metadata is harder than writing them
  • No existing spec discipline — the approach requires the spec to be correct and complete, which is an organisational property, not a technical one

The approach pays off with: multiple studies sharing a standard, a stable company data standard, a define.xml requirement, and a team large enough that consistency between programmers is a real problem.

Common mistakes

Mistake Consequence Fix
Spec and code maintained separately Drift, discovered in QC Code reads the spec
Hard-coding labels alongside a spec Two sources of truth Apply from spec only
Not checking data against the spec Non-conformant output compare_to_spec() at the end
Over-engineering the metadata More time on the framework than the study Start small, extend when it hurts
eval(parse()) on uncontrolled input Arbitrary code execution Controlled files only; prefer a check vocabulary
Spec in Word, converted by hand Transcription errors Spec in Excel or CSV, read directly

Exercise 5.1 — Apply a specification

Given an ADSL dataset and a metacore specification, write the pipeline that drops unspecified variables, checks controlled terminology, orders columns, applies labels and lengths, and reports any discrepancy before writing the XPT.

Show solution 
library(metacore); library(metatools); library(xportr)
library(dplyr); library(purrr); library(cli)

apply_spec <- function(data, spec, dataset_name, xpt_path = NULL) {

  cli_h2("Applying specification for {dataset_name}")

  # --- 1. Structural conformance ------------------------------------------
  spec_vars <- spec$var_spec$variable
  data_vars <- names(data)

  extra   <- setdiff(data_vars, spec_vars)
  missing <- setdiff(spec_vars, data_vars)

  if (length(extra) > 0) {
    cli_alert_info("Dropping {length(extra)} unspecified variable{?s}: {.var {extra}}")
  }
  if (length(missing) > 0) {
    cli_abort(c(
      "{dataset_name} is missing {length(missing)} specified variable{?s}.",
      "x" = "{.var {missing}}",
      "i" = "Either derive them or update the specification."
    ))
  }

  out <- data |>
    drop_unspec_vars(spec) |>
    check_variables(spec)

  # --- 2. Controlled terminology ------------------------------------------
  ct_result <- tryCatch({
    check_ct_data(out, spec, na_acceptable = TRUE)
    "PASS"
  }, error = function(e) {
    cli_alert_danger("Controlled terminology check failed:")
    cli_bullets(c("x" = conditionMessage(e)))
    "FAIL"
  })
  if (ct_result == "FAIL") cli_abort("Resolve CT violations before proceeding.")

  # --- 3. Order and sort ---------------------------------------------------
  out <- out |> order_cols(spec) |> sort_by_key(spec)

  # --- 4. Length conformance, BEFORE writing -------------------------------
  overlong <- map(names(out), function(v) {
    if (!is.character(out[[v]])) return(NULL)
    spec_len <- spec$var_spec$length[spec$var_spec$variable == v]
    if (length(spec_len) == 0 || is.na(spec_len)) return(NULL)
    actual <- max(c(0L, nchar(out[[v]], type = "bytes")), na.rm = TRUE)
    if (actual > spec_len) {
      tibble::tibble(variable = v, spec_length = spec_len, actual_length = actual,
                     example = out[[v]][which.max(nchar(out[[v]], type = "bytes"))])
    }
  }) |> compact() |> list_rbind()

  if (nrow(overlong) > 0) {
    cli_abort(c(
      "{nrow(overlong)} variable{?s} exceed{?s/} the specified length.",
      set_names(
        sprintf("%s: spec %d, actual %d (%s)",
                overlong$variable, overlong$spec_length, overlong$actual_length,
                substr(overlong$example, 1, 40)),
        rep("x", nrow(overlong))
      ),
      "i" = "Truncate the values or increase the length in the specification."
    ))
  }

  # --- 5. Apply transport metadata ----------------------------------------
  out <- out |>
    xportr_type(spec)     |>
    xportr_length(spec)   |>
    xportr_label(spec)    |>
    xportr_format(spec)   |>
    xportr_df_label(spec)

  # --- 6. Write -------------------------------------------------------------
  if (!is.null(xpt_path)) {
    xportr_write(out, xpt_path, strict_checks = TRUE)
    cli_alert_success("Wrote {.path {basename(xpt_path)}} ({nrow(out)} records)")
  }

  invisible(out)
}

Used:

meta <- spec_to_metacore("metadata/adam_spec.xlsx")

adsl_final <- apply_spec(
  adsl,
  select_dataset(meta, "ADSL"),
  "ADSL",
  xpt_path = "data/submission/adsl.xpt"
)

The design decision worth explaining: checking lengths before xportr_length(), not after. xportr_length() sets the length attribute; if a value is longer than the specified length, the truncation happens silently at write time and you lose data without noticing. Checking first turns that into an error with the offending value shown.

The other choice: extra variables are dropped with an info message, but missing variables are a hard error. That asymmetry is right — an extra working variable is untidy, a missing specified variable means the deliverable is incomplete.

Exercise 5.2 — Generate a data dictionary from produced data

Write a function that takes a list of final ADaM datasets and produces an Excel workbook documenting every dataset and variable — the reverse direction, useful for reconciling against the spec or bootstrapping one.

Show solution 
library(dplyr); library(purrr); library(openxlsx)

build_data_dictionary <- function(datasets, path) {

  # --- Dataset-level sheet ------------------------------------------------
  ds_sheet <- imap(datasets, function(d, nm) {
    tibble::tibble(
      Dataset     = nm,
      Label       = attr(d, "label") %||% NA_character_,
      Records     = nrow(d),
      Variables   = ncol(d),
      Subjects    = if ("USUBJID" %in% names(d)) n_distinct(d$USUBJID) else NA_integer_,
      Structure   = case_when(
        "USUBJID" %in% names(d) && !anyDuplicated(d$USUBJID) ~ "One record per subject",
        all(c("PARAMCD", "AVISITN") %in% names(d))           ~ "One record per subject, parameter and visit",
        .default                                              ~ "One record per subject and occurrence"
      )
    )
  }) |> list_rbind()

  # --- Variable-level sheet -----------------------------------------------
  var_sheet <- imap(datasets, function(d, nm) {
    tibble::tibble(
      Dataset  = nm,
      Order    = seq_along(d),
      Variable = names(d),
      Label    = map_chr(d, ~ attr(.x, "label") %||% NA_character_),
      Type     = map_chr(d, function(x) {
        cls <- class(x)[1]
        case_when(cls %in% c("numeric", "double")  ~ "float",
                  cls == "integer"                 ~ "integer",
                  cls == "Date"                    ~ "integer",
                  cls %in% c("POSIXct", "POSIXt")  ~ "datetime",
                  .default                          ~ "text")
      }),
      Format   = map_chr(d, ~ attr(.x, "format.sas") %||% NA_character_),
      Length   = map_int(d, function(x) {
        if (is.character(x)) max(c(1L, nchar(x, type = "bytes")), na.rm = TRUE)
        else 8L
      }),
      N_Missing  = map_int(d, ~ sum(is.na(.x))),
      Pct_Missing = round(100 * map_int(d, ~ sum(is.na(.x))) / max(nrow(d), 1), 1),
      N_Distinct = map_int(d, ~ n_distinct(.x, na.rm = TRUE))
    )
  }) |> list_rbind()

  # --- Value-level sheet: distinct values for low-cardinality variables ---
  value_sheet <- imap(datasets, function(d, nm) {
    keep <- names(d)[map_lgl(d, function(x) {
      (is.character(x) || is.factor(x)) && n_distinct(x, na.rm = TRUE) <= 25
    })]

    map(keep, function(v) {
      d |>
        count(.data[[v]], name = "N") |>
        filter(!is.na(.data[[v]])) |>
        transmute(Dataset = nm, Variable = v,
                  Value = as.character(.data[[v]]), N,
                  Pct = round(100 * N / nrow(d), 1))
    }) |> list_rbind()
  }) |> list_rbind()

  # --- Write --------------------------------------------------------------
  hdr <- createStyle(textDecoration = "bold", fgFill = "#16355e",
                     fontColour = "white", halign = "left")

  wb <- createWorkbook()
  walk2(
    list(ds_sheet, var_sheet, value_sheet),
    c("Datasets", "Variables", "ValueLevel"),
    function(dat, sheet) {
      addWorksheet(wb, sheet)
      writeData(wb, sheet, dat, headerStyle = hdr)
      freezePane(wb, sheet, firstRow = TRUE)
      setColWidths(wb, sheet, cols = seq_along(dat), widths = "auto")
      addFilter(wb, sheet, rows = 1, cols = seq_along(dat))
    }
  )

  # Provenance
  addWorksheet(wb, "About")
  writeData(wb, "About", tibble::tibble(
    Item = c("Generated", "By", "R version", "Git commit", "Datasets"),
    Value = c(format(Sys.time()), Sys.info()[["user"]], R.version.string,
              tryCatch(system("git rev-parse HEAD", intern = TRUE),
                       error = \(e) "unknown"),
              paste(names(datasets), collapse = ", "))
  ), headerStyle = hdr)

  saveWorkbook(wb, path, overwrite = TRUE)
  cli::cli_alert_success("Data dictionary written to {.path {path}}")
  invisible(list(datasets = ds_sheet, variables = var_sheet, values = value_sheet))
}

Used:

adam <- list(
  ADSL = readRDS("data/adam/adsl.rds"),
  ADAE = readRDS("data/adam/adae.rds"),
  ADLB = readRDS("data/adam/adlb.rds")
)

build_data_dictionary(adam, "output/data_dictionary.xlsx")

Two uses for this:

  1. Reconciliation. Diff the Variables sheet against the specification’s variable sheet. Any row that differs is either a code bug or a spec that needs updating — and you want to know which before the QC programmer finds it.
  2. Bootstrapping a spec. For a legacy study with no machine-readable specification, this produces a first draft in the right shape, which a human then corrects. Far faster than starting from a blank template.
The Length calculation deserves note: it reports the actual maximum byte width, which is what the XPT file will need. A spec that says 200 for a variable whose longest value is 12 wastes space in the transport file, and Pinnacle 21 will raise it as a finding.

Recap

  • The specification is data the code reads, not a document a human transcribes
  • metacore loads a spec into six related tables; metatools applies it
  • Standard pipeline: derive with admiral, conform with metatools, write with xportr
  • Check lengths before xportr_length() — truncation at write time is silent
  • Metadata can drive categorisations, parameter lookups and validation checks
  • Compare the produced dataset back against the spec, and fail on any difference
  • The approach pays off across multiple studies, not on a single small one

Next: TLF generation.

Back to top