Clinical study folder structure

Lesson 1 — Clinical Programming with R

Lesson 1 of 12 Intermediate ~60 min

Learning objectives

  • Design a study repository that supports reproducibility and audit
  • Separate source data, derived data and outputs correctly
  • Apply naming conventions that survive a regulatory review
  • Manage environments per study with renv and containers
  • Decide what belongs in a study repository and what belongs in a package

What the structure has to support

A clinical study repository is not just an organised folder. It has to answer, years later and to someone who was not there:

  • Which program produced this table?
  • Which version of which dataset did it read?
  • Which package versions were installed?
  • Who changed what, when, and who reviewed it?
  • Can the whole thing be re-run and produce identical output?

Every structural decision below exists to make one of those answerable.

A working layout

abc101/
├── README.md
├── abc101.Rproj
├── renv.lock
├── .Rprofile
├── .gitignore
│
├── metadata/                   # specifications, as data
│   ├── adam_spec.xlsx
│   ├── sdtm_spec.xlsx
│   ├── define_2_1.xml
│   └── controlled_terminology.csv
│
├── data/
│   ├── raw/                    # source — READ ONLY, never in Git
│   │   ├── sdtm/
│   │   │   ├── dm.sas7bdat
│   │   │   ├── ae.sas7bdat
│   │   │   └── lb.sas7bdat
│   │   └── external/
│   │       └── lab_normal_ranges.xlsx
│   ├── adam/                   # derived — regenerable, not in Git
│   │   ├── adsl.rds
│   │   ├── adae.rds
│   │   └── adlb.rds
│   └── submission/             # xpt files for transfer
│       ├── adsl.xpt
│       └── adae.xpt
│
├── programs/
│   ├── adam/
│   │   ├── ad_adsl.R
│   │   ├── ad_adae.R
│   │   └── ad_adlb.R
│   ├── tlf/
│   │   ├── t_14_1_1_demographics.R
│   │   ├── t_14_3_1_ae_summary.R
│   │   ├── l_16_2_1_disposition.R
│   │   └── f_14_2_1_km_pfs.R
│   ├── qc/
│   │   ├── qc_adsl.R
│   │   └── qc_adae.R
│   └── run_all.R
│
├── R/                          # study-specific reusable functions
│   ├── derivations.R
│   ├── formats.R
│   └── table_helpers.R
│
├── tests/
│   └── testthat/
│       ├── test-derivations.R
│       └── test-formats.R
│
├── output/                     # regenerable, not in Git
│   ├── tables/
│   ├── listings/
│   ├── figures/
│   └── logs/
│
└── docs/
    ├── sap.pdf
    ├── programming_conventions.md
    └── validation_plan.md

The rules that make it work

1. data/raw/ is read-only. No program writes to it. If a program needs to fix something in source data, that fix is a documented derivation in data/adam/, not an edit to the source. Enforce it at the filesystem level if you can:

chmod -R a-w data/raw/

2. Everything except data/raw/ and source code is regenerable. Delete data/adam/ and output/ and re-run run_all.R — you should get identical files. If you cannot, there is hidden state somewhere.

3. Never commit patient data.

# .gitignore
data/raw/
data/adam/
data/submission/
output/
*.sas7bdat
*.xpt
*.rds
renv/library/
.Rhistory
.RData
.Rproj.user

Commit metadata/ — the specifications are the crown jewels and they contain no patient data.

4. Programs are numbered or ordered by dependency, and run_all.R makes the order explicit rather than implicit.

Naming conventions

Consistency matters more than the specific choice, but these are conventional:

Type Pattern Example
ADaM program ad_<dataset>.R ad_adsl.R
SDTM program sdtm_<domain>.R sdtm_ae.R
Table t_<number>.R t_14_1_1_demographics.R
Listing l_<number>.R l_16_2_4_ae.R
Figure f_<number>.R f_14_2_1_km_pfs.R
QC program qc_<original>.R qc_adsl.R
Output file matches the program t_14_1_1_demographics.rtf

The output filename matching the program name is the single most useful convention: given a table in a submission package, you can find the program that made it without looking anything up.

Variable naming follows CDISC — uppercase for standard variables (USUBJID, AVAL, TRTEMFL), and it is worth being strict about it because define.xml and the transport format both care.

The program header

Every program starts with a header. In a paper-based world this was the audit trail; it is still where a reviewer looks first.

#-------------------------------------------------------------------------------
# Study:        ABC-101
# Program:      ad_adsl.R
# Purpose:      Create ADSL (subject-level analysis dataset)
#
# Input:        data/raw/sdtm/dm.sas7bdat
#               data/raw/sdtm/ex.sas7bdat
#               data/raw/sdtm/ds.sas7bdat
#               metadata/adam_spec.xlsx (sheet: ADSL)
#
# Output:       data/adam/adsl.rds
#               data/submission/adsl.xpt
#
# Specification: ADaM Specification v2.0, section 3.1
# SAP reference: Section 6.1 (analysis populations)
#
# Author:       R Gaduputi
# Date:         2026-07-28
#
# Revision history:
#   2026-07-28  RG   Initial version
#   2026-08-14  RG   Added PPROTFL per protocol amendment 2 (CR-118)
#-------------------------------------------------------------------------------
TipGenerate the header, do not type it

A header that is copy-pasted and half-edited is worse than none — it will eventually claim the wrong author and the wrong inputs. Use an RStudio snippet or a usethis-style function:

new_adam_program <- function(dataset) {
  path <- file.path("programs", "adam", paste0("ad_", tolower(dataset), ".R"))
  writeLines(glue::glue(read_template("adam_header")), path)
  file.edit(path)
}

Paths

Never setwd(). Use here so a program works from the console, from run_all.R, from a Quarto document and from a scheduled job:

library(here)

adsl <- readRDS(here("data", "adam", "adsl.rds"))
saveRDS(out, here("data", "adam", "adae.rds"))

Better still, centralise the paths so a change of environment is one edit:

# R/paths.R
study_paths <- function(env = Sys.getenv("STUDY_ENV", "development")) {
  base <- switch(env,
    development = here::here(),
    validation  = "/mnt/validation/abc101",
    production  = "/mnt/production/abc101"
  )

  list(
    sdtm       = file.path(base, "data", "raw", "sdtm"),
    adam       = file.path(base, "data", "adam"),
    submission = file.path(base, "data", "submission"),
    output     = file.path(base, "output"),
    metadata   = file.path(base, "metadata")
  )
}

p <- study_paths()
dm <- haven::read_sas(file.path(p$sdtm, "dm.sas7bdat"))

Environments

Three environments, promoted in order:

Environment Data Who Purpose
Development Test or blinded extract Programmers Writing and debugging
Validation Full, snapshot QC programmers Independent verification
Production Locked database Automated run Deliverable output

The promotion is by Git tag, not by copying files:

git tag -a v1.0-dbl1 -m "Database lock 1 — TLF delivery 2026-09-15"
git push --tags

Production runs a checkout of a tag, in a defined environment, and archives the log.

Reproducible environments

renv per study, always:

renv::init()
renv::snapshot()

Commit renv.lock. It pins package versions but not R itself, the OS, or system libraries. For a deliverable, add a container:

FROM rocker/r-ver:4.4.1

RUN apt-get update && apt-get install -y --no-install-recommends \
    libxml2-dev libssl-dev libcurl4-openssl-dev \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /study
COPY renv.lock renv.lock
RUN R -q -e "install.packages('renv', repos='https://cloud.r-project.org')" \
 && R -q -e "renv::restore(repos = c(CRAN = 'https://packagemanager.posit.co/cran/2026-07-01'))"

COPY . .
CMD ["Rscript", "programs/run_all.R"]

The dated Posit Package Manager snapshot is what makes a rebuild in 2029 install the same versions as today. Archive the built image alongside the deliverable.

run_all.R

The single entry point. It makes the dependency order explicit and records what happened.

#-------------------------------------------------------------------------------
# Study:   ABC-101
# Program: run_all.R
# Purpose: Execute the full analysis pipeline in dependency order
#-------------------------------------------------------------------------------

library(here)
library(cli)

start_time <- Sys.time()
log_file   <- here("output", "logs",
                   format(start_time, "run_%Y%m%d_%H%M%S.log"))
dir.create(dirname(log_file), recursive = TRUE, showWarnings = FALSE)

con <- file(log_file, open = "wt")
sink(con, split = TRUE)
sink(con, type = "message")
on.exit({ sink(type = "message"); sink(); close(con) }, add = TRUE)

cli_h1("ABC-101 analysis run")
cli_alert_info("Started:     {format(start_time)}")
cli_alert_info("R version:   {R.version.string}")
cli_alert_info("Git commit:  {system('git rev-parse HEAD', intern = TRUE)}")
cli_alert_info("Git status:  {if (length(system('git status --porcelain', intern = TRUE)) == 0)
                               'clean' else 'DIRTY — output is not reproducible'}")
cli_alert_info("User:        {Sys.info()[['user']]}")

programs <- c(
  # ADaM — order matters, ADSL first
  "programs/adam/ad_adsl.R",
  "programs/adam/ad_adae.R",
  "programs/adam/ad_adlb.R",
  "programs/adam/ad_advs.R",
  # Tables, listings, figures — independent of each other
  "programs/tlf/t_14_1_1_demographics.R",
  "programs/tlf/t_14_3_1_ae_summary.R",
  "programs/tlf/l_16_2_1_disposition.R",
  "programs/tlf/f_14_2_1_km_pfs.R"
)

results <- vapply(programs, function(prog) {
  cli_h2(prog)
  t0 <- Sys.time()

  status <- tryCatch({
    source(here(prog), echo = FALSE)
    "SUCCESS"
  }, error = function(e) {
    cli_alert_danger("FAILED: {conditionMessage(e)}")
    "FAILED"
  })

  cli_alert_info("{status} in {round(difftime(Sys.time(), t0, units = 'secs'), 1)}s")
  status
}, character(1))

cli_h1("Summary")
print(data.frame(program = basename(programs), status = unname(results)))

writeLines(capture.output(sessionInfo()),
           here("output", "logs", "sessionInfo.txt"))

if (any(results == "FAILED")) {
  cli_abort("{sum(results == 'FAILED')} program{?s} failed.")
}
cli_alert_success("Run complete in {round(difftime(Sys.time(), start_time, units = 'mins'), 1)} minutes")

The git status check is small and valuable: a run from a dirty working directory cannot be reproduced from the commit, and recording that fact in the log prevents a false claim later.

Study repository or package?

Put in the study repository Put in a shared package
Study-specific derivations Standard derivations used across studies
TLF programs TLF templates and helpers
Study metadata Company standard metadata structures
run_all.R Utilities, formats, validation checks
Anything referencing this protocol Anything protocol-agnostic

The mature pattern: a company standards package (versioned, tested, validated once) plus thin study repositories that call it. That is exactly how admiral is designed to be used — see Pharmaverse workflows.

Common mistakes

Mistake Consequence Fix
Writing to data/raw/ Source data no longer matches the database Make it read-only
Committing .sas7bdat Patient data in Git history, permanently .gitignore before the first commit
setwd() in programs Only runs on one machine here()
No run_all.R Nobody knows the correct order Explicit pipeline
renv.lock not committed Different versions on the validation server Commit it
Output filenames unrelated to programs Cannot trace a table to its source Match them
Running from a dirty working tree Output not reproducible from any commit Check and record git status

Exercise 1.1 — Design a study repository

Set up the folder structure for a study with 4 ADaM datasets, 12 tables, 6 listings and 3 figures, with independent QC programming. Write the .gitignore and the skeleton of run_all.R.

Show solution 
# setup_study.R — run once
library(fs)

dirs <- c(
  "metadata",
  "data/raw/sdtm", "data/raw/external",
  "data/adam", "data/submission",
  "programs/adam", "programs/tlf", "programs/qc", "programs/utils",
  "R",
  "tests/testthat",
  "output/tables", "output/listings", "output/figures", "output/logs",
  "output/qc",
  "docs"
)
dir_create(dirs, recurse = TRUE)

# Placeholder so Git tracks the empty directories
walk(c("data/raw/sdtm", "data/adam", "output/tables", "output/logs"),
     ~ file_create(path(.x, ".gitkeep")))

.gitignore:

# ---- Patient data: never commit -------------------------------------------
data/raw/**
!data/raw/**/.gitkeep
data/adam/**
!data/adam/.gitkeep
data/submission/**
*.sas7bdat
*.sas7bcat
*.xpt
*.rds
*.parquet

# ---- Regenerable output ---------------------------------------------------
output/**
!output/**/.gitkeep

# ---- R and RStudio --------------------------------------------------------
.Rproj.user/
.Rhistory
.RData
.Ruserdata
renv/library/
renv/staging/
renv/local/

# ---- OS -------------------------------------------------------------------
.DS_Store
Thumbs.db

Note the !...gitkeep negations — without them the directory structure is not in the repository and a fresh clone fails on the first saveRDS().

programs/run_all.R:

#-------------------------------------------------------------------------------
# Study:   ABC-101
# Program: run_all.R
# Purpose: Execute the full pipeline. Production runs invoke only this.
#
# Usage:   Rscript programs/run_all.R              # everything
#          Rscript programs/run_all.R adam         # ADaM only
#          Rscript programs/run_all.R tlf          # TLF only
#          Rscript programs/run_all.R qc           # QC only
#-------------------------------------------------------------------------------

library(here); library(cli); library(purrr)

args  <- commandArgs(trailingOnly = TRUE)
stage <- if (length(args) == 0) "all" else args[1]

# --- Pipeline definition: dependency order is explicit ----------------------

pipeline <- list(
  adam = c(
    "programs/adam/ad_adsl.R",     # must be first — others read it
    "programs/adam/ad_adae.R",
    "programs/adam/ad_adlb.R",
    "programs/adam/ad_adtte.R"
  ),
  tlf = c(
    sprintf("programs/tlf/t_14_%d.R", 1:12),
    sprintf("programs/tlf/l_16_%d.R", 1:6),
    sprintf("programs/tlf/f_14_%d.R", 1:3)
  ),
  qc = c(
    "programs/qc/qc_adsl.R",
    "programs/qc/qc_adae.R",
    "programs/qc/qc_adlb.R",
    "programs/qc/qc_adtte.R",
    "programs/qc/qc_compare_all.R"  # runs diffdf across every pair
  )
)

to_run <- if (stage == "all") unlist(pipeline, use.names = FALSE)
          else pipeline[[stage]] %||% cli_abort("Unknown stage: {stage}")

# --- Logging ----------------------------------------------------------------

t_start  <- Sys.time()
log_path <- here("output", "logs",
                 format(t_start, paste0(stage, "_%Y%m%d_%H%M%S.log")))
dir.create(dirname(log_path), recursive = TRUE, showWarnings = FALSE)

con <- file(log_path, open = "wt")
sink(con, split = TRUE); sink(con, type = "message")
on.exit({ sink(type = "message"); sink(); close(con) }, add = TRUE)

git_sha   <- tryCatch(system("git rev-parse HEAD", intern = TRUE), error = \(e) "unknown")
git_dirty <- length(tryCatch(system("git status --porcelain", intern = TRUE),
                             error = \(e) character())) > 0

cli_h1("ABC-101 — stage: {stage}")
cli_alert_info("Started    {format(t_start)}")
cli_alert_info("R          {R.version.string}")
cli_alert_info("Commit     {git_sha}")
if (git_dirty) {
  cli_alert_warning("Working tree is DIRTY — this run is not reproducible from a commit")
}
cli_alert_info("Programs   {length(to_run)}")

# --- Execute ----------------------------------------------------------------

results <- map(to_run, function(prog) {
  cli_h2(basename(prog))
  t0 <- Sys.time()

  err <- NULL
  status <- tryCatch({
    source(here(prog), echo = FALSE, local = new.env())   # isolate each program
    "SUCCESS"
  }, error = function(e) {
    err <<- conditionMessage(e)
    cli_alert_danger("{conditionMessage(e)}")
    "FAILED"
  })

  elapsed <- as.numeric(difftime(Sys.time(), t0, units = "secs"))
  cli_alert_info("{status} ({round(elapsed, 1)}s)")

  tibble::tibble(program = basename(prog), status = status,
                 elapsed_s = round(elapsed, 1), error = err %||% NA_character_)
}) |> purrr::list_rbind()

# --- Summary ----------------------------------------------------------------

cli_h1("Summary")
print(as.data.frame(results), row.names = FALSE)

readr::write_csv(results, here("output", "logs",
                               format(t_start, paste0(stage, "_%Y%m%d_%H%M%S_summary.csv"))))
writeLines(capture.output(sessionInfo()),
           here("output", "logs", "sessionInfo.txt"))

n_fail <- sum(results$status == "FAILED")
if (n_fail > 0) {
  cli_abort("{n_fail} program{?s} failed. See {.path {log_path}}.")
}
cli_alert_success("Complete in {round(difftime(Sys.time(), t_start, units = 'mins'), 1)} min")

Three choices worth noting:

  • local = new.env() on each source(). Without it, objects created by one program remain visible to the next, and a program that accidentally depends on a leftover object works in run_all.R and fails when run alone.
  • Stage arguments. Re-running only the TLFs after a formatting fix is a daily need; forcing a full ADaM rebuild each time wastes hours.
  • The summary CSV alongside the log. The log is for a human; the CSV is what a validation report can quote.

Exercise 1.2 — Make a run reproducible

A colleague delivers a table and cannot say which data version produced it. Design a mechanism that makes this impossible in future, with no extra manual work.

Show solution

Stamp provenance automatically at three points: on the data, on the output, and in the log.

1. A provenance record, captured once per run

# R/provenance.R

capture_provenance <- function() {
  sdtm_dir <- study_paths()$sdtm
  files <- list.files(sdtm_dir, pattern = "\\.(sas7bdat|xpt)$", full.names = TRUE)

  list(
    run_id      = format(Sys.time(), "%Y%m%dT%H%M%S"),
    run_time    = Sys.time(),
    user        = Sys.info()[["user"]],
    host        = Sys.info()[["nodename"]],
    r_version   = R.version.string,
    git_sha     = tryCatch(system("git rev-parse HEAD", intern = TRUE),
                           error = \(e) NA_character_),
    git_tag     = tryCatch(system("git describe --tags --exact-match 2>/dev/null",
                                  intern = TRUE), error = \(e) NA_character_),
    git_clean   = length(tryCatch(system("git status --porcelain", intern = TRUE),
                                  error = \(e) character())) == 0,
    renv_hash   = tools::md5sum("renv.lock")[[1]],
    inputs      = tibble::tibble(
      file     = basename(files),
      size     = file.size(files),
      modified = file.mtime(files),
      md5      = unname(tools::md5sum(files))
    )
  )
}

The MD5 of each source file is the part that actually answers the question. A filename and a date can both be identical between two different extracts; a checksum cannot.

2. Attach it to every derived dataset

save_adam <- function(data, name, prov = getOption("study.provenance")) {
  attr(data, "provenance") <- prov

  path <- file.path(study_paths()$adam, paste0(tolower(name), ".rds"))
  saveRDS(data, path)

  # A sidecar JSON, readable without R
  jsonlite::write_json(
    c(prov, list(dataset = name, n_rows = nrow(data), n_cols = ncol(data))),
    sub("\\.rds$", "_provenance.json", path),
    auto_unbox = TRUE, pretty = TRUE
  )

  invisible(data)
}

3. Stamp it on the output itself

provenance_footnote <- function(prov = getOption("study.provenance")) {
  sprintf(
    "Program: %s | Commit: %s%s | Run: %s | R %s",
    prov$program %||% "unknown",
    substr(prov$git_sha, 1, 7),
    if (isFALSE(prov$git_clean)) " (UNCOMMITTED CHANGES)" else "",
    format(prov$run_time, "%d%b%Y %H:%M"),
    getRversion()
  )
}

tbl |>
  r2rtf::rtf_source(provenance_footnote()) |>
  r2rtf::rtf_encode() |>
  r2rtf::write_rtf(out_path)

4. Wire it in once, in run_all.R

options(study.provenance = capture_provenance())

jsonlite::write_json(
  getOption("study.provenance"),
  here("output", "logs", sprintf("provenance_%s.json",
                                 getOption("study.provenance")$run_id)),
  auto_unbox = TRUE, pretty = TRUE
)

Now every table carries its own answer in the source footnote, every derived dataset has a sidecar JSON, and the run log has the full record — and no programmer has to remember to do anything.

The uncomfortable extra

if (isFALSE(prov$git_clean)) " (UNCOMMITTED CHANGES)" prints a warning directly onto the output. That is deliberate. A table produced from uncommitted code cannot be reproduced, and making that visible on the artefact is far more effective than a note in a log nobody reads. For a production run you would go further and refuse to proceed:

if (!prov$git_clean && Sys.getenv("STUDY_ENV") == "production") {
  cli::cli_abort("Production runs require a clean working tree.")
}

Recap

  • The structure exists to answer “what produced this?” years later
  • data/raw/ is read-only; everything else must be regenerable
  • Never commit patient data — .gitignore before the first commit
  • Output filenames match program names, so a table traces to its source
  • here() and a central path function; never setwd()
  • renv.lock plus a container with a dated repository snapshot
  • run_all.R is the single entry point and records commit, versions and checksums
  • Study-specific code in the study repo; reusable code in a versioned package

Next: Reading SAS7BDAT and XPT files.

Back to top