Case study 1: Generating and prioritizing PGS models

This scenario illustrates a common PolyGenius workflow for Alzheimer’s disease (AD): generate a set of candidate PGS models from one GWAS source, and benchmark them in cohort data to prioritize the best-performing model.

Show setup code

library(dplyr)
library(data.table)
library(ggplot2)
library(patchwork)
library(colorspace)

library(PolyGenius)

Generate candidate AD models

We generate the PGS models from the Bellenguez et al. summary statistics (Bellenguez et al. 2022) from IEU OpenGWAS (Elsworth et al. 2020), to which we apply multiple different algorithms. The ClumpingThresholding runs use the EUR reference panel from the 1000 Genomes Project (The 1000 Genomes Project Consortium 2015). The key point is that a single generate$models(...) call can request several algorithms at once, while the execution engine resolves and reuses shared intermediate tasks such as summary-statistics retrieval, reference-panel preparation, clumping, and LD-matrix construction.

# Generate AD-PGS models
models <- generate$models(
  sources = generate$sources$opengwas("ebi-a-GCST90027158"),
  algorithms = c(
    generate$algorithms$ClumpingThresholding(pval = c(5e-7, 5e-8, 5e-9), reference.panel = "EUR", variant.space = "HapMap3+"),
    generate$algorithms$LDpred2(reference.panel = "EUR", ld.size = 3000, ld.thr = .01, ncores = 8, variant.space = "HapMap3+"),
    generate$algorithms$lassosum2(reference.panel = "EUR", ld.size = 3000, ld.thr = .01, ncores = 8, variant.space = "HapMap3+")
  )
)

models <- generate$models(
  sources = generate$sources$opengwas("ebi-a-GCST90027158"),
  algorithms = c(
    generate$algorithms$ClumpingThresholding(pval = c(5e-7, 5e-8, 5e-9)),
    generate$algorithms$LDpred2(ld.size = 3000, ld.thr = .01, ncores = 8),
    generate$algorithms$lassosum2(ld.size = 3000, ld.thr = .01, ncores = 8),
  ),
  reference.panel = "EUR", 
  variant.space = "HapMap3+"
)

models

Evaluate candidate models

Once the candidate models have been generated, we create a single PolyGeniusData object that combines the models, the cohort genotypes, and the phenotype table required for benchmarking. Here the evaluation cohort consists of two genotype datasets, the Amsterdam Dementia Cohort (ADC) (Flier and Scheltens 2018) and the Longitudinal Aging Study Amsterdam (LASA) (Hoogendijk et al. 2016, 2025). Since both cohort datasets are defined with the GRCh38 genome build, we first liftover the generated models.

liftover(models, "GRCh38")

data <- PolyGeniusData(
  name = "evaluate",
  models = models,
  genotypes = c(
    GenotypeInfo(path = "resources/cohorts/ADC/genotypes", format = "pfile", build = "GRCh38", name = "ADC"),
    GenotypeInfo(path = "resources/cohorts/LASA/genotypes", format = "pfile", build = "GRCh38", name = "LASA")
  ),
  obsm = list(
    phenotypes = rbind(
      fread("resources/cohorts/ADC/phenotypes/phenotypes.csv")[, c("demented", "sex")],
      fread("resources/cohorts/LASA/phenotypes/phenotypes.csv")[, c("demented", "sex")]
    ) %>% mutate(demented = factor(demented, 0:1))
  )
)

data

We then run evaluate$benchmark(...) to orchestrate a joint model-assessment workflow which: - Assesses models’ ability to distinguish between demented and non demented samples - calibration - similarity - compare - risk strata - incremental

using dementia status as the outcome and sex as the baseline covariate.

# results <- evaluate$benchmark(
#   on = data,
#   outcome = demented,
#   baseline.covariates = sex
# )

results1 <- evaluate$similarity(on = data)
results2 <- evaluate$discrimination(on = data, outcome = demented)
results3 <- evaluate$incremental(on = data, outcome = demented, baseline.covariates = sex)

results <- merge(merge(results1, results2), results3)

corr.p <- visualize$evaluate$similarity$heatmap(results, 
  center.zero = TRUE, 
  symmetrize.range = TRUE, 
  border = TRUE, 
  name = "Pearson correlation", 
  column_names_rot = 45,
  width = data$n_mod * unit(8, "mm"),
  height = data$n_mod * unit(8, "mm"))

ROC.p <- visualize$evaluate$discrimination$roc(results) + 
  scale_color_discrete_sequential("BuPu") + 
  theme(legend.position = c(.75, .2))

Evaluation reporting and visualization

The benchmark result can now be visualized from several complementary angles. The plotting block below collects overview, discrimination, calibration, comparison, similarity, incremental-value, and risk-stratification views that together support model ranking and prioritization.

pdf("Rplots.pdf", width = 7, height = 5)

print(visualize$evaluate$overview$metric.heatmap(results))
print(visualize$evaluate$overview$leaderboard(results))

print(visualize$evaluate$discrimination$curve(results))
print(
  visualize$evaluate$discrimination$roc(results) +
    ggplot2::theme(legend.position = "right") +
    ggplot2::theme_classic()
)
print(visualize$evaluate$discrimination$pr(results))
print(visualize$evaluate$discrimination$confusion(results))
print(visualize$evaluate$discrimination$distribution(results))
print(visualize$evaluate$discrimination$forest(results))

print(visualize$evaluate$calibration$curve(results))
print(visualize$evaluate$calibration$slope.intercept(results))

print(visualize$evaluate$compare$delta.forest(results))
print(visualize$evaluate$compare$delta.heatmap(results))

print(visualize$evaluate$similarity$heatmap(results, border = TRUE))

print(visualize$evaluate$incremental$delta(results))

print(visualize$evaluate$risk.strata$profile(results))
print(visualize$evaluate$risk.strata$lift(results))

dev.off()

References

Bellenguez, Celine, Fatma Kucukali, Iris E. Jansen, et al. 2022. “New Insights into the Genetic Etiology of Alzheimer’s Disease and Related Dementias.” Nature Genetics 54 (4): 412–36. https://doi.org/10.1038/s41588-022-01024-z.

Elsworth, Benjamin, Milo Lyon, Lucia Alexander, et al. 2020. “The MRC IEU OpenGWAS Data Infrastructure.” bioRxiv. https://doi.org/10.1101/2020.08.10.244293.

Flier, Wiesje M. van der, and Philip Scheltens. 2018. “Amsterdam Dementia Cohort: Performing Research to Optimize Care.” Journal of Alzheimer’s Disease 62 (3): 1091–111. https://doi.org/10.3233/JAD-170850.

Hoogendijk, Emiel O., Dorly J. H. Deeg, Jan Poppelaars, et al. 2016. “The Longitudinal Aging Study Amsterdam: Cohort Update 2016 and Major Findings.” European Journal of Epidemiology 31 (9): 927–45. https://doi.org/10.1007/s10654-016-0192-0.

Hoogendijk, Emiel O., Natasja M. van Schoor, Yuwei Qi, et al. 2025. “The Longitudinal Aging Study Amsterdam: Design and Cohort Update 2025.” European Journal of Epidemiology 40 (6): 705–20. https://doi.org/10.1007/s10654-025-01238-5.

The 1000 Genomes Project Consortium. 2015. “A Global Reference for Human Genetic Variation.” Nature 526 (7571): 68–74. https://doi.org/10.1038/nature15393.