Key messages
° Many NTCP models have been developed to predict unwanted effects following radiotherapy in head and neck cancer patients but most of them have not been sufficiently externally validated, that is, tested with patients who were not involved in the original model developmental study, to know how well they really predict the unwanted effects.
° For models tested in two or more studies in addition to their original model development studies, the quality of testing and reporting of their results was generally poor, so it is difficult to know how useful they might be.
° More and better designed studies are required to investigate this issue in the area of head and neck cancer.
How can we decide on the likelihood of having unwanted effects as a result of treatment?
The likelihood of having unwanted effects as a result of radiotherapy can be calculated by using so-called NTCP models. NTCP models calculate the risk of radiation-induced side effects based on information from the patient, their disease, and their treatment.
What did we want to find out?
Radiotherapy is the mainstay of treatment of patients with head and neck cancer. However, radiotherapy exposes healthy, sometimes crucial, parts of the head and neck region to radiation. This may result in damage to these normal organs, e.g. disturbed saliva production, which may have important consequences for the quality of life of head and neck cancer patients treated with radiotherapy. To reach an optimal balance between tumour control and preventing radiation-induced side effects, normal tissue complication probability (NTCP) models can be helpful. These models predict the risk of radiation-induced side effects based on information from the patient, their disease, and their treatment. There have been a substantial number of NTCP models for patients with head and neck cancer. We wanted to find out what the quality of study design, conduct, and analysis (i.e. risk of bias) is, and how well these models can predict the risk of radiation-induced side effects.
What did we do?
We searched for studies that developed and/or validated NTCP models in patients with head and neck cancer.
What did we find?
In most of the 592 models developed from 140,767 patients in 143 identified articles, the quality of the models was not sufficient; and it has not been investigated how well they perform in new patients for 81% of these models. For the remaining 19% of the models, 152 external validations were found in 34,304 patients from 41 articles. There were only nine models with two or more external validations. The models were able to distinguish patients with and without the outcome well, but it was often unclear whether their predictions were in line with what was observed, because the latter was not always assessed and/or reported. Overall, the quality of most of these studies was low.
How up to date is the review?
The evidence is current to 8th January 2024.
Read the full abstract
Radiotherapy is the mainstay of treatment for head and neck cancer (HNC) but may induce various side effects on surrounding normal tissues. To reach an optimal balance between tumour control and toxicity prevention, normal tissue complication probability (NTCP) models have been reported to predict the risk of radiation-induced side effects in patients with HNC. However, the quality of study design, conduct, and analysis (i.e. risk of bias (ROB)), as well as the predictive performance of these models, remains to be evaluated.
Objectives
To identify, describe and appraise NTCP models to predict the risk of radiation-induced side effects in patients with HNC.
Search strategy
We searched Ovid MEDLINE, Embase and the World Health Organization International Clinical Trials Registry Platform from conception to January 2024. In addition, we screened references cited in the retrieved articles.
Selection criteria
Two review authors independently included articles reporting on the development and external validation of NTCP models to predict any type of radiation-induced side effects in patients with HNC.
Data collection and analysis
One reviewer extracted data from each article based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies and assessed their applicability and ROB using the Prediction model ROB Assessment Tool, while another reviewer carefully verified the results. For models externally validated at least twice for the same outcome as their original developmental study, we performed qualitative analyses of the model performance. The GRADE system was not applied, since it has not been established for reviews of prognostic model studies.
Main results
Amongst 592 models developed from 143 articles, including 140,767 HNC patients, only 49 (8%) models from six articles were judged to have low ROB and low concerns for applicability. No external validation was performed for 480 models (81%). For the remaining 112 models and six additional models which were not eligible for the present review, 152 external validations were performed in 34,304 patients with HNC in 41 articles. The results of models externally validated at least twice are discussed below.
Models for xerostomia
Amongst 275 models for xerostomia, two models were externally validated at least twice.
The Beetz 2012b model for xerostomia six months after radiotherapy was validated in two studies. C-statistics ranged from 0.70 to 0.74. Calibration performance was reported in one study. One validation study was rated as having low ROB in all domains, while the other was rated as having high ROB in the analysis domain.
The Cavallo 2021 model for acute xerostomia during radiotherapy for patients with nasopharyngeal cancer was externally validated in the same study, using two different types of cohorts. C-statistics ranged from 0.68 to 0.73 and calibration plots were reported in both cohorts. Both validations were rated as having unclear ROB in the participants' domain because no detailed information about recruiting was provided.
Models for dysphagia
Amongst 86 models for dysphagia, two models were externally validated at least twice.
The Christianen 2012 model for dysphagia six months after radiotherapy was validated in five studies. C-statistics ranged from 0.66 to 0.75. Calibration performance was assessed in all of them, while four of them were rated as having high ROB in the analysis domain due to the small sample size.
The Wopken 2014b model for tube feeding dependence six months after radiotherapy was validated in three external validation studies. C-statistics ranged from 0.79 to 0.95, while calibration was evaluated in all studies. Due to the small size of the validation datasets, they were judged as having high ROB in the analysis domain.
Models for hypothyroidism
Of 66 models for hypothyroidism, two models were externally validated at least twice. In addition, there was another model which was not originally developed for patients with HNC, but validated in this domain.
The Boomsma 2012 for hypothyroidism within two years after radiotherapy was externally validated in two studies. C-statistics ranged from 0.64 to 0.74, while only one study reported its calibration performance. Both validation studies were rated as having high ROB in the analysis domain.
The Ronjom 2013 model for radiation-induced hypothyroidism was validated in three studies. C-statistics ranged from 0.65 to 0.69 and calibration plots were reported in only one study. Two validation studies were judged as having high and the other was rated as having unclear ROB in the analysis domain.
The Cella 2012 model was originally developed to predict radiation-induced hypothyroidism in patients with Hodgkin’s lymphoma. In two validation studies in patients with HNC, c-statistics ranged from 0.65 to 0.68, but calibration performance was not reported. One validation study was rated as having a high ROB and the other was rated as being unclear in the analysis domain.
Models for temporal lobe injury
Amongst six models for temporal lobe injury, two were externally validated at least twice.
The OuYang 2023 model, using deep learning in patients with nasopharyngeal cancer, was validated in the same paper using two different cohorts. C-statistics ranged from 0.80 to 0.82, while calibration performance was assessed in both cohorts. Both validations were judged as having low ROB in all domains.
The Wen 2021 model was developed to predict temporal lobe injury in newly diagnosed nasopharyngeal cancer patients. The model was validated by OuYang 2023 using two cohorts. C-statistics ranged from 0.77 to 0.79, while calibration performance was not reported. Both validations were judged as having unclear ROB in the analysis domain.
Models for outcomes related to hoarseness, fatigue, nausea-vomiting, throat pain, aspiration
No models were externally validated at least twice.
Authors' conclusions
Amongst 592 developed models, a limited number had adequate quality. Only one-fifth were externally validated, of which, only nine models at least twice. These nine models showed acceptable discriminative performance at external validation. However, their calibration performance was not always reported. Furthermore, most validation studies were judged as having high ROB, mainly due to problems in the analysis domain. In conclusion, this review shows the need for more external validation studies before the implementation of developed models in clinical practice and improvement of the quality of conducting and reporting of prediction model studies.
