Marc Cerou (1,2), Marylore Chenel (2), Emmanuelle Comets (1,3)
(1) Université de Paris, IAME, INSERM, Paris, France (2) Institut de Recherches Internationales, Suresnes, France (3) Université Rennes 1, CIC1414, INSERM, Rennes, France
Objectives: Model evaluation is now a well-established pivotal step of model building and qualification in pharmacometrics, with an array of diagnostic tools for standard PK/PD analyses including simulation-based prediction discrepancies (pd/npd) and prediction distribution errors (pde/npde) [1]. Pharmacometric models however have diversified to include data such as TTE, for which TTE-npde have been proposed recently [2], or repeated measurements of ordered categorical variables (e.g. score ACR, pain, nausea, depression) [3]. For this last type of data, adequate estimation algorithms have been proposed but diagnostic tools are scant.
In the present paper, we develop and evaluate npd for ordinal data.
Methods: A prediction discrepancy (pd) is the quantile of an observation in its marginal predictive distribution. As the data is ordinal, the cumulative distribution is a step function. If all subjects are from the same population and the design is homogenous with the same sample times, the pd at any given time can take only one of K discrete values (pi0=0, pi1, …, piK=1). In this very simple case, at any given time we can compare the observed proportions of pd falling in each category to the predicted proportions assuming the model is correct, using a chi-squared test for multinomial distributions. In longitudinal studies, this test is repeated for the npd obtained at each time. To account for both multiple tests and the correlation between measurements in the same subject, we correct the test statistics by obtaining their distribution under model M.
When covariates enter the model or when the design is not homogenous, the discrete values depend on the predictive distribution at each observation and can no longer be combined as proportions. Instead, we propose in this more general case to transform these pd to a continuous variable, sampling the pd for the kth category in the interval [piK-1,piK] (letting pi0=0). The resulting pd are then expected to follow a uniform distribution. We take the inverse normal function to transform them into npd; npd follow a normal distribution and we use the global test proposed by Brendel et al., combining a variance, mean and normality test [4]. However, the resulting npd still involve within subject correlations, so again we build the distribution of the test statistic through simulations under model M to obtain a corrected threshold for the test.
We evaluated the performance of the npd in a simulation study with repeated binary data. The model and parameters were inspired by the setting in [5], where a logistic model was used to describe the probability of response to treatment over time. In order to compare the global test for npd with the proportion test, we considered only one treatment arm, with 3 sample sizes (N=50, 100, 200) and 4 optimised sampling times.
Several scenarios were run, considering either model misspecifications on fixed or variability parameters, or misspecifications in the structural model (testing log-linear, quadratic, and exponential functions). For each scenario, we generated 200 datasets, and we approximated the predictive distribution using 1000 replicates. For each dataset we compared the proportion test for pd and the global test based on npd.
Results: In all scenarios, we found controlled type I error both for npd and proportion tests. As expected, the power increased with an increase of the sample size.
The power increased as the difference from the true parameter value increased, except with a misspecification on the inter-individual variability for the intercept in the logistic model, where the power was not different from 5% with both tests. On the other hand, the power to detect a misspecification in the structural model was very low with the global test on npd and only slightly higher with the proportion test.
Conclusions: The npd were extended to ordinal data and showed adequate type I error in a simulation study on repeated binary data. Power to detect model misspecification in the structural model was low, but this was expected as the models and their respective parameter values were chosen in order to get similar profiles compared to the true model.
With homogenous designs we proposed a more powerful test based on comparing the observed proportions of each set of pd with their value under the model, but this approach is not practicable in most cases, which makes the npd more useful in general settings.
References:
[1] Nguyen THT, Mouksassi M, Holford N, Al-Huniti N, Freedman I, Hooker AC, et al (2017). Model evaluation of continuous data pharmacometric models: metrics and graphics. CPT Pharmacometrics Syst Pharmacol 6:87–109.
[2] Cerou M, Lavielle M, Brendel K, Chenel M, Comets E (2018). Development and performance of npde for the evaluation of time-to-event models. Pharm Res 35:30.
[3] Niebecker R, Maas H, Staab A, Freiwald M, Karlsson MO (2019). Modeling exposure-driven adverse event time courses in oncology exemplified by afatinib. CPT Pharmacometrics Syst Pharmacol 8:230-239.
[4] Brendel K, Comets E, Laffont C, Laveille C, Mentré F (2006). Metrics for external model evaluation with an application to the population pharmacokinetics of gliclazide. Pharm Res 23:2036-49.
[5] Seurat J, Nguyen TT, Mentré F (2019). Robust designs accounting for model uncertainty in longitudinal studies with binary outcomes. Stat Methods Med Res (in press).
Reference: PAGE () Abstr 9495 [www.page-meeting.org/?abstract=9495]
Poster: Methodology - Model Evaluation