A Machine Learning Approach to Design an Efficient Selective Screening of Mild Cognitive Impairment

The methodology applied in this study has been previously published⁵ in work carried out at the University CEU Cardenal Herrera together with community pharmacies in the region of Valencia (Spain) associated with the Spanish Society of Family and Community Pharmacy (SEFAC). This current study was reviewed and approved by the Research Ethics Committee at the Universidad CEU Cardenal Herrera (approval no. CEI11/001) in March 2011. All individuals involved in the study gave their written informed consent to participation in accordance with the Declaration of Helsinki.

1. Selection of factors associated with mild cognitive impairment

1. Search for terms related to MCI for use in screening Cochrane Systematic Reviews (e.g., cognitive impairment, dementia, risk factors, etc.).
2. Search for terms for which there is some evidence of a relationship with cognitive deterioration or dementia published in the PubMed database; these include demographic factors (sex, age, education level, and economic status), social factors (cognitive and social activities), chronic pathologies (cholesterol, depression, hypertension, diabetes, and obesity), and lifestyle behaviors (alcohol consumption, smoking habit, diet, physical activity, and sleep hours).
3. Calculate the odds ratio for qualitative variables or Cohen's d effect size for quantitative variables¹⁵. Select the variables with larger effect sizes for cognitive deterioration or dementia for use in elaborating a questionnaire.

2. Design of the questionnaires

1. Design a questionnaire to collect information about the selected variables, following the guidelines provided by Nardi¹⁶. For instance, the variables used in Climent et al.⁵ were demographic (age, weight, and height [measured with standardized procedures using calibrated scales and stadiometers], sex, education level, and employment type), lifestyle (physical exercise, reading, time spent sleeping overnight and during the day, puzzles, games, TV consumption time, and tobacco and alcohol consumption), and chronic pathologies (hypertension, hyperlipidemia, and diabetes). In addition, record the presence or absence of depression, which is frequently associated with cognitive deterioration.
2. Design a pharmacotherapy follow-up sheet to report all the drugs consumed by the participants at the time of the interview, as in Climent et al.⁵, which used Dader's method¹⁷ to design this sheet.

3. Selection of tests for MCI screening

1. Determine all the tests used to screen for MCI that could be administered by primary healthcare workers (e.g., pharmacists). Reject any tests that must be administered by a specialist. Some of the tests that fulfill these conditions are the Short Portable Mental State Questionnaire (SPMSQ)¹⁸, Mini Mental State Examination (MMSE)¹⁹, Memory Impairment Screen (MIS)²⁰, Picture Memory Impairment Screen (PMIS)²¹, Montreal Cognitive Assessment (MoCA)²², Saint Louis University Mental Status (SLUMS)²³, and Quick Mild Cognitive Impairment (Qmci)²⁴. An exhaustive review of each MCI test is available in Cullen et al.²⁵.
2. Search for a good estimation of the test sensitivities and specificities in the scientific literature.
3. Estimate the time required to administer these tests to healthy individuals.
4. Consider the basic patient characteristics required for completion of these tests. For example, a minimum education level may be necessary because many MCI tests are not suitable for illiterate participants. A set of MCI screening tests is usually applied to increase sensitivity; however, the minimum number of tests must be quickly administered by pharmacists if the final selective screening is intended for a large population. Climent et al.⁵ assessed MCI using the MMSE and SPMSQ tests, with the latter being suitable for the large number of individuals who lived through the Spanish civil war who are illiterate.
   1. Use a variant of the SPMSQ by Pfeiffer¹⁸ was validated in Spanish by Martínez de la Iglesia²⁶. This test has a maximum score of 10 and the cut-off point for establishing cognitive impairment is 3 or more errors (4 or more for illiterate individuals). This test takes between 8 and 10 minutes to complete.
   2. Use a NORMACODERM version of the MMSE validated for Spanish speakers by Blesa²⁷ by adapting the original version by Folstein¹⁹. This screening test has a maximum score of 30 and is corrected according to the patients' years of schooling and ages. Participants who score less than or equal to 24 are considered as MCI cases. The MMSE is a measure of general cognitive function and includes orientation to time and place, written and spoken language, attention span, calculation, and memory. It was administered to all the participants in this study because it is a very short test which takes only around 5 minutes to complete.

4. Subject recruitment

1. Find pharmacists willing to recruit non-institutionalized people to form the study population. The mentioned study by Climent et al.⁵ included people aged 65 years or more who went regularly to the pharmacy and who agreed to participate in this study. Exclude patients with any difficulty in performing these evaluation tests (e.g., because of blindness, deafness, etc.) or who were already being treated for dementia.
2. Provide the participating pharmacists with informed consent forms, which must be completed by every individual taking part in the study. This consent form specifies the title of the research, the objectives of the project, a comprehensible explanation of all the procedures that the participant would take part in, the absence of specific risks, the confidentiality of all the collected data, and the right to withdraw from the study for any reason at any time.
3. Train the pharmacists to administer structured personal interviews to the participants, which should last approximately half an hour per person. Collect data for 1 year and send all the forms to the researchers responsible for data protection in the study. Subsequently follow-up with the patients for 3 months.
4. Instruct the pharmacists how to identify a probable MCI case using MCI tests. Based on Climent et al.⁵ we used SPMSQ scores of 4 or more points (for illiterate participants) or 3 or more points for the other participants and scores of 24 points or less were used in the corrected MMSE test.
5. Instruct pharmacists how to refer MCI cases to a medical specialist (a neurologist) for their clinical diagnosis-the last step in the flow chart used in this research study (Figure 1).

Figure 1: Flowchart of the research study and the proposed selective screening. The left side represents the initial study whose data were analyzed with machine-learning techniques to propose the selective screening for early detection of MCI shown in the right panel. This figure was modified from Climent³⁴. Please click here to view a larger version of this figure.

5. Pharmacist researcher training

1. Contact specialists to organize sessions for training the participating pharmacists in basic knowledge related to cognitive impairment and in managing its screening tools, for instance, the SPMSQ and MMSE.
2. Ensure that the participating pharmacists are aware of the procedures, data collection protocol, and all the possible issues related to data protection. Inform them that the project was approved by a Research Ethics Committee and of the importance of the consent form according to the Declaration of Helsinki.
   NOTE: To perform the study described by Climent et al.⁵, workshops were held at the Official College of Pharmacists and the Cardenal-Herrera CEU University (UCH-CEU), and covered the following: MCI and dementia; diagnostic approaches to MCI and management of the SPMQP and MMSE (taught by the Neurology Service at La Plana Hospital in Castellón); project presentation and explanation of the methodology by senior community pharmacist researchers; and health education and cognitive training by researchers from the Department of Pharmacy at the UCH-CEU University.

6. Study design

1. Calculate a sample size to assess the feasibility of the project. Because this was an observational study, a larger sample will produce more effective tools. There are two ways to determine sample size: one is based on the estimation of the prevalence and the other is more precise, taking into account effect sizes.
   1. Calculate an accurate estimation of the prevalence of the condition in the population
      
      
      
      where α is the significance level, p₀ is the initial estimation and error is the maximum error expected with a 100(1 – α)% confidence.
   2. According to the effect sizes found in the literature for each factor, use tools like the pwr package in R to estimate how much power is required to detect differences^15,28.
      NOTE: For instance, in our study⁵ we designed the first proposal with an error of 3% at 95% confidence and an initial estimation of the prevalence of MCI at 15% in the population aged 65 years or older, resulting in an estimated required sample size of 541 individuals.

7. Interdisciplinary communication network, pharmacists, primary healthcare physicians, and specialists

1. Design letters to communicate information about the project to the healthcare centers involved.
2. Explain to participating pharmacists how to inform their assigned physicians about the results of the screenings through a letter to the primary healthcare center.
3. Send written communications to the medical coordinators of the healthcare centers related to the participating pharmacies and to the Neurology Services of the hospitals to which they are assigned.
4. Contact participating neurologists to find out each patient's definitive diagnosis obtained via specific tests undertaken by specialized healthcare providers. Before this, primary healthcare providers should carry out the following protocol, as summarized by the clinical guidelines (Figure 2).

Figure 2: Protocol for primary healthcare action. An example of primary healthcare actions that should be considered for early MCI detection before the patient is referred for a medical diagnosis by specialists. Please click here to view a larger version of this figure.

8. Statistical analysis and preprocessing

NOTE: Before applying machine-learning techniques a preparatory step is required to transform the original data into a new data set according to the final study objective and the procedures to be applied. For this transformation, several things should be considered, including the characteristics of the algorithms. This is because some of them are sensitive to a lack of variability or sharing of information across columns, although the algorithms used to generate decision trees are particularly robust against these problems. This initial phase aims to categorize qualitative variables and gather values with enough cases for each variable. For efficient screening it is important to choose variables whose acquisition is proven to be easy and accurate. Participants are selected by a short interview in which the algorithms used were constrained to a white-box model, making it easy to check the criteria used to decide if the individual should take the test. We suggest using the rpart²⁹ package in R software for these algorithms, and implementing recursive partitioning.

1. Collect all the forms from the participating pharmacies and convert them into a table in which every column is a variable and every participating individual is a row.
2. Assign an identification number to each participant. Save the identification number and contact information in a different document so that it is not used by the machine-learning algorithm.
3. Generate variables to classify whether each drug the patient takes corresponds to second or third ATC³⁰ (Anatomical Therapeutic Chemical) level codes, according to the active principal ingredients on the pharmacotherapy follow-up sheet.
4. Perform an initial descriptive analysis.
   1. For every ordinal variable, choose an adequate contrast for the variable. For categorical variables, select the value considered as the baseline.
   2. For categorical variables, calculate a univariate logistic regression with a response variable for screening for MCI. Analyze the outcome of the regression with a contingency table, the p-value, sample odds ratio, and the 95% confidence interval of the odds ratio.
   3. For quantitative variables, calculate the mean, standard deviation, coefficient of logistic regression, and the 95% confidence interval of their coefficients.
5. Reject variables with missing (unavailable) values, considering these variables difficult to accurately collect.
6. Select only variables for which there is at least one statistically significant category (α < 0.01) according to the logistic regression analysis. The outcome of this step produces a reduced data set compared to the initial one.

9. Algorithms to create a decision tree

NOTE: Machine-learning algorithms must be properly parameterized to predict which individuals are likely to have a positive MCI test result. One of the main problems while screening for a condition is that the original data is expected to be imbalanced (i.e., few positive cases compared to the negative ones). To get models with balanced data we used a technique called down-sampling, or random sampling, to equalize the frequency with that of the lowest frequency class³¹. Efficient screening also requires reducing the number of false negatives as much as possible (i.e., increasing the sensitivity of the selection of participants suffering from MCI). One of the techniques used to achieve a greater sensitivity is the introduction of penalties in the calculation of Gini's impurity index (i.e., the index used by the algorithm to select the best split for the decision tree)³².

1. Generate a training and test data set with 80% and 20% of the whole data set, respectively using the createDataPartition function in the caret library³³.
2. Apply the algorithms used to generate decision trees to the training data set. Use the train again function in the caret library³³. The following steps are different parameters of the function; for instance, the tree used in this paper was generated with rpart²⁹ (method = "rpart"), but other algorithms are available.
   1. Select the 'down sampling' sampling method and introduce the sampling = "down" parameter into the caret.
   2. Set the prior probabilities for both classes.
   3. Provide a loss matrix with the Gini's impurity index penalties applied in order to focus on the increasing sensitivity.
   4. For every parameter in the algorithm, choose an appropriate grid of values.
   5. Use a cross-validation estimation of the receiver operating curve (ROC) values to select the best models within the parameter grid.
3. Calculate a confusion matrix and the area under the ROC curve (AUC) for the test set prediction to assess the true performance of the model.