Multimedia Battery for Assessment of Cognitive and Basic Skills in Mathematics (BM-PROMA)

Cristina Rodr&#237;guez; Juan E. Jim&#233;nez; Sara C. de Le&#243;n; Isaac Marco

doi:10.3791/62288

JoVE Journal > Behavior

Behavior

Multimedia Battery for Assessment of Cognitive and Basic Skills in Mathematics (BM-PROMA)

Published: August 28, 2021

doi:

10.3791/62288

Cristina Rodríguez*^1,2, Juan E. Jiménez*¹, Sara C. de León*¹, Isaac Marco*¹

¹Faculty of Psychology and Speech Therapy,Universidad de La Laguna, ²Faculty of Education,Universidad Católica del Maule

Summary

BM-PROMA is a valid and reliable multimedia diagnostic tool that can provide a complete cognitive profile of children with mathematical learning disabilities.

Abstract

Learning mathematics is a complex process that requires the development of multiple domain-general and domain-specific skills. It is therefore not unexpected that many children struggle to stay at grade level, and this becomes especially difficult when several abilities from both domains are impaired, as in the case of mathematical learning disabilities (MLD). Surprisingly, although MLD is one of the most common neurodevelopmental disorders affecting schoolchildren, most of the diagnostic instruments available do not include assessment of domain-general and domain-specific skills. Furthermore, very few are computerized. To the best of our knowledge, there is no tool with these features for Spanish-speaking children. The purpose of this study was to describe the protocol for the diagnosis of Spanish MLD children using the BM-PROMA multimedia battery. BM-PROMA facilitates the evaluation of both skill domains, and the 12 tasks included for this purpose are empirically evidence-based. The strong internal consistency of BM-PROMA and its multidimensional internal structure are demonstrated. BM-PROMA proves to be an appropriate tool for diagnosing children with MLD during primary education. It provides a broad cognitive profile for the child, which will be relevant not only for diagnosis but also for individualized instructional planning.

Introduction

One of the crucial objectives of primary education is the acquisition of mathematical skills. This knowledge is highly relevant, as we all use mathematics in our everyday lives, for example, to calculate change given at the supermarket¹^,². As such, the consequences of poor mathematical performance go beyond the academic. At the social level, a strong prevalence of poor mathematical performance within the population constitutes a cost to society. There is evidence that improvement of poor numerical skills in the population leads to significant savings for a country³. There are also negative consequences at an individual level. For example, those who show a low level of mathematical skills present poor professional development (e.g., higher rates of employment in poorly paid manual occupations and higher unemployment)⁴^,⁵^,⁶, frequently report negative socio-emotional responses towards academics (e.g., anxiety, low motivation towards academics)⁷^, ⁸, and tend to present poorer mental and physical health than their peers with average mathematical achievement⁹. Students with mathematical learning disabilities (MLD) show very poor performance that persists over time¹⁰^,¹¹^,¹². As such, they are more likely to suffer the consequences mentioned above, especially if these are not promptly diagnosed¹³.

MLD is a neurobiological disorder characterized by severe impairment in terms of learning basic numerical skills despite adequate intellectual capacity and schooling¹⁴. Although this definition is widely accepted, the instruments and criteria for its identification are still under discussion¹⁵. An excellent illustration of the absence of a universal agreement regarding MLD diagnosis is the variety of reported prevalence rates, ranging from 3 to 10%¹⁶^,¹⁷^,¹⁸^,¹⁹^,²⁰^,²¹. This difficulty in diagnosis stems from the complexity of mathematical knowledge, which requires that a combination of multiple domain-general and domain-specific skills be learned²²^,²³. Children with MLD show very different cognitive profiles, with a broad constellation of deficits¹⁴^, ²⁴^,²⁵^,²⁶^,²⁷. In this regard, it is suggested that the need for multidimensional assessment by means of tasks involving different numerical representations (i.e., verbal, Arabic, analogic) and arithmetical skills¹¹.

In primary school, symptoms of MLD are diverse. In terms of domain-specific skills, it is consistently found that many MLD students show difficulties in basic numerical skills, such as quickly and accurately recognizing Arabic numerals²⁸^,²⁹^,³⁰, comparing magnitudes³¹^,³², or representing numbers on the number line³³^,³⁴. Primary school children have also shown difficulty in understanding conceptual knowledge, such as place value³⁵, arithmetic knowledge³⁶, or ordinality measured through ordered sequences³⁷. Regarding domain-general skills, particular focus has been put on the role of working memory³⁸^,³⁹ and language⁴⁰ in the development of mathematical skills in children with and without MLD. In relation to working memory, the results suggest that students with MLD show a deficit in the central executive, especially when required to manipulate numerical information⁴¹^,⁴². A deficit in visuospatial short-term memory has also been frequently reported in children with MLD⁴³^,⁴⁴. Language skills have been found to be a prerequisite for learning numeracy skills, especially those that involve high verbal processing demand⁷. For example, phonological processing skills [e.g., phonological awareness and Rapid Automatized Naming (RAN)] are closely linked to those basic skills learned in primary school, such as numerical processing or arithmetic calculation³⁹^,⁴⁵^,⁴⁶^,⁴⁷. Here, it has been demonstrated that variations in phonological awareness and RAN are associated with individual differences in numeracy skills that involve managing verbal code⁴²^,⁴⁸. In light of the complex profile of children with MLD, a diagnostic tool should ideally include tasks that assess both domain-general and domain-specific skills, which are reported as being more frequently deficient in these children.

In recent years, several paper-and-pencil screening tools for MLD have been developed. Those most commonly used with Spanish primary school children are a) Evamat-Batería para la Evaluación de la Competencia Matemática (Battery for the Evaluation of Mathematical Competency)⁴⁹; b) Tedi-Math: A Test for Diagnostic Assessment of Mathematical Disabilities (Spanish adaptation)⁵⁰; c) Test de Evaluación Matemática Temprana de Utrecht (TEMT-U)⁵¹^,⁵², the Spanish version of the Utrecht Early Numeracy Test⁵³; and d) Test of early math abilities (TEMA-3)⁵⁴. These instruments measure many of the domain-specific skills mentioned above; however, none of them assess domain-general skills. Another limitation of these instruments – and of paper-and-pencil tools in general – is that they cannot provide information regarding the accuracy and automaticity with which each item is processed. This would only be possible with a computerized battery. However, very few applications have been developed for dyscalculia diagnosis. The first computerized tool designed to identify children (aged 6 to 14) with MLD was the Dyscalculia Screener⁵⁵. A few years later, the web-based DyscalculiUm⁵⁶ was developed with the same purpose but focused on adults and learners in post-16 education. Although still limited, there has been growing interest in computerized tool design for the diagnosis of MLD in recent years⁵⁷^,⁵⁸^,⁵⁹^,⁶⁰. None of the tools mentioned have been standardized for Spanish children, and only one of them – the MathPro Test⁵⁷– includes domain-general skill evaluation. Given the importance of identifying children with low mathematical achievement, especially those with MLD, and in the absence of computerized instruments for the Spanish population, we present a multimedia evaluation protocol that includes both domain-general and domain-specific skills.

Protocol

This protocol was conducted in accordance with the guidelines provided by the Comité de Ética de la Investigación y Bienestar Animal (Research Ethics and Animal Welfare Committee, CEIBA), Universidad de La Laguna.

NOTE: The Batería multimedia para la evaluación de habilidades cognitivas y básicas en matemáticas [Multimedia Battery for Assessment of Cognitive and Basic Skills in Mathematics (BM-PROMA)]⁶¹ was developed using Unity 2.0 Professional Edition and the SQLITE Database Engine. BM-PROMA includes 12 subtests: 8 to assess domain-specific skills and 4 to evaluate domain-general processes. For each subtest, provide instructions orally by an animated humanoid robot and precede the testing phase with a demonstration and two training trials. The application protocol for each task is presented below with an example.

1. Experimental setup

Use the following inclusion criteria: children in primary education between second and sixth grade; native speakers of Spanish.
Use the following exclusion criteria: children with a history of neurological,intellectual, or sensory deficits.
Install the Multimedia Battery for Assessment of Cognitive and Basic Skills in Mathematics. BM-PROMA is distributed using a single file. This file is an automated installer that allows the user to select the installation destination. The installer detects previous versions of the tool and warns the user about possible data loss due to overwriting. The installation creates shortcuts in the Windows 'Start' menu. Additionally, the installer provides a batch file (known as a .bat file in Windows) to automate the database backup process. The tool runs in full screen mode at a resolution of 800×600 pixels. The tool cannot run in windowed mode.
1. Before a student can be assessed, add their data to the student database. Once the child has been registered, select them by clicking on the relevant entry in the student list. Tasks are selected at random by the examiner or the child. Tasks begin as soon as the examiner or the child clicks on them. When the task is complete, the tool returns to the task selection menu. Tasks completed by the student are no longer visible in the menu. Once the session has begun, there are no breaks between tasks.
2. Test children grades 2 and 3 in three half hour sessions and children grades 4 to 6 in two 45-minute sessions. Hold the sessions on different days. Administer the BM-PROMA in a quiet room. Have students use a headset to listen to the instructions and to record their oral responses; the examiner also uses headphones to monitor the tasks. In some cases, the examiner must record the outcome of the task using the mouse; in others, the student uses the mouse to complete the task and responses are recorded automatically.
Demonstration and training trials.For all tasks, precede the test stage with instructions (the robot orally presents the instructions for the task), modelling (the robot models the task step by step with an example), and practice trials (children are allowed up to two practice trials with feedback).

2. Domain-specific subtests

Missing number (Figure 1)
1. In this task, ask children to name the missing number from a series of 4 single- and two-digit numbers presented horizontally.
2. Have the robot say the following: "In this game, you have to say aloud the name of the missing number: two, four, six, eight, and (pause) ten. So, the missing number is ten. Now, try it on your own".
3. Present a total of 18 series: 6 in numerically ascending order (the numbers in the series increase in value as a given magnitude is added to the previous number), 6 in numerically descending order (the numbers in the series decrease in value as a given magnitude is subtracted from the previous number), and 6 in numerically hierarchical ascending order (more than one arithmetic operation is needed to solve them, in this case, multiplication and addition). The examiner uses the mouse buttons to record whether the answer is correct.
4. Calculate the score based on the total number of correct responses.
Two-digit number comparison (Figure 2)
1. In this task, present 40 pairs of two-digit numbers on the computer screen.
2. Have the robot say "In this game, look carefully at these two numbers. You must choose the biggest number. To do so, you must compare the two numbers and say aloud the name of the biggest one. Look at these two numbers. Thirty-seven is bigger than twenty-one. So, I will say /thirty-seven/. Try to complete the task as quickly as possible without getting it wrong. Now, try it on your own".
3. Require children to say aloud the numerically larger of each pair. A voice key registered the child's reaction time (RT), after which the examiner used the mouse buttons to record whether the answer was correct.
  NOTE: Following previous studies⁶²^,⁶³, unit-decade compatibility (compatible vs. incompatible) and decade and unit distance (small [1-3] vs. large [4-8]) were manipulated.
4. Calculate the score based on the RT of those stimuli that were solved correctly.
Reading numbers (Figure 3)
1. Present 30 Arabic numbers (10 single-digit numbers, 10 two-digit numbers, and 10 three-digit numbers) one at a time on the computer screen.
2. Have the robot say "In this game, you have to name aloud the numbers that appear on the screen. Look at this number. Here you have to say /twelve/, because that is the name of the number on the screen. Try to complete the task as quickly as possible without getting it wrong. Now, try it on your own".
3. Ask the child to read them aloud as quickly as possible without making mistakes. A voice key registered the child's RT, after which the examiner used the mouse buttons to record whether the answer was correct.
4. Calculate the score based on the RT of those stimuli that were read correctly.
Place value (Figure 4)
1. Measure the students' knowledge of the Arabic number system. Display 12 two-digit Arabic numbers in the center of the computer screen, with one answer option located in each corner of the screen (four options in total). Each option was a quantity represented by tiny blocks of units and blocks of tens (ten units grouped into a single block). For each item, only one of the four options was correct. The incorrect options were made up of representations that coincided with the correct option in a) the ten; b) the unit; or c) both the ten and the unit, but reversed (e.g., for the number "15", the incorrect options represented 12, 35 and 51).
2. Have the robot say "In this game, we have a number and four pictures. You have to click on the picture that represents the number correctly. I will choose the first one, because the bar equals a ten, and the squares equal five units. Now, try it on your own".
3. Calculate the score based the total number of correct responses.
Number line 0-100 and 0-1000 tasks (Figure 5)
NOTE: Use computerized adaptations of the paper-and-pencil original⁶⁴.
1. In this task, have children position a given number on a 15-cm number line using the computer mouse. For the first 20 items, the value at the left end of the line was 0 and the value at the right end was 100. For the following 22 items, the value at the right end was 1000.
2. Present the following items on the 0-100 line: 2, 3, 7, 11, 14, 18, 23, 37, 41, 45, 56, 60, 67, 71, 75, 86, 89, 91, 95 and 99.
3. Have the robot say "In this game, you have to put the number where you think it should go. Look at this line. It starts at zero and ends at one hundred. You have to put number fifty here. To do so, click and hold on the red line under the number and drag it to the correct place. Do you know why I dropped the number here? It is in the middle, because fifty is half of one hundred. Now, try it on your own".
4. Following the original task, oversample the numbers at the low end of the distribution, with 7 numbers between 0 and 30. The items presented for the 0-1000 line were: 2, 11, 67, 99, 106, 162, 221, 325, 388, 450, 492, 511, 591, 643, 677, 755, 799, 815, 867, 910 and 988. Values below 100 were oversampled, as in the aforementioned study.
5. Calculate the score based on the absolute value of the percentage error (|Estimate – Estimated Quantity / Scale of Estimates|).
Arithmetic fact retrieval (Figure 6)
1. Ask children to solve 66 single-digit arithmetic problems, consisting of 24 additions, 24 multiplications, and 18 subtractions presented in separate blocks. Exclude tie problems (e.g., 3+3) and problems containing 0 or 1 as operand or answer.
2. Have the robot say "In this game, you have to solve the computations in your head. In the first one, the correct answer is three. Solve the task in silence and tell me the answer aloud. Try to solve the task as quickly as possible without getting it wrong. Now, try it on your own.
3. Present problems one at a time horizontally on the computer screen. Responses were verbal. A voice key registered the child's RT, after which the examiner used the mouse buttons to record whether the answer was correct.
4. Calculate the score based on the RT of those stimuli that were solved correctly.
Arithmetic principles (Figure 7)
1. Present 24 pairs of related two-digit operations (12 pairs of additions and 12 pairs of multiplications). In each pair, one item was solved correctly and the other was unsolved (e.g., 5+5=10 → 5+6=?).
2. Have the robot say "In this game, you have to say aloud the result of the second operation. Look carefully at both computations. The first one has already been solved, but the second one still needs to be solved. Five plus five equals ten, then five plus six equals eleven. When I tell you to start, solve the task in silence and then say the answer aloud. Try to solve the task as quickly as possible without getting it wrong. Now, try it on your own".
3. Ask children to say aloud the result of the unsolved operation. A voice key registered the child's reaction time (RT), after which the examiner used the mouse buttons to record whether the answer was correct.
4. Calculate the score based on the RT of those stimuli that were solved correctly.

3. Domain-general subtests

Counting span (Figure 8)
NOTE: This task is an adaptation of the Working Memory-Counting task⁶⁵.
1. Have the children count aloud the number of yellow dots on a series of cards with yellow and blue dots. Ask them to recall the number of yellow dots on each card in the set.
2. Have the robot say "In this game, we have some cards. Each card has blue and yellow dots. You have to count and remember the number of yellow dots on each card. First, we are going to count how many yellow dots there are on the first card. There are two yellow dots on the card. Then we will count all the yellow dots on the second card. There are eight yellow dots on the card. Now, as there were two yellow dots on the first card and eight yellow dots on the second card, you have to say aloud the numbers two and eight. Now, try it on your own".
3. Increase set length from 2 to 5 cards and give the children three attempts to move to the next level of difficulty. The examiner uses the mouse buttons to record whether the answer is correct.
4. End the test when a child fails to correctly recall two sets at a given difficulty level.
Rapid Automatized Naming – Letter (RAN-L) (Figure 9)
NOTE: This task is an adaptation of the technique called Rapid Automatized Naming⁶⁶. RAN-L consists of a series of five letters presented in five rows and 10 columns on the computer screen.
1. Ask the child to name the letters as quickly as possible from left to right and from top to bottom. Provide ten practice items in a chart consisting of two rows and five columns.
2. Have the robot say "In this game, you have to name the letters that appear on the screen. It does not matter if they are repeated. So, we have to say: /a/, /c/, /v/, /n/, /a/, /n/, /c/, /c/, /v/, /v/. Try to name the letters as quickly as possible from left to right and from top to bottom. Now, try it on your own".
3. Use the time spent to name all 50 letters as the score. To normalize the score distribution, convert the scores to the number of letters per minute.
Visuospatial working memory (Figure 10)
NOTE: This task is a computerized adaptation of the Corsi block-tapping task⁶⁷.
1. Show a 3×3 board in the center of the screen. In each trial, sequentially flash certain blocks on and off.
2. Ask the child to repeat the sequence in the correct order by clicking on the blocks that had changed color. In 50% of cases, ask them to do so in the same order, and in the other 50% in reverse order.
3. Have the robot say "In this game, you will see that some of the squares light up. You have to remember which squares lit up and the order in which they did so. Then you have to press the squares in the same order to repeat the sequence. Now, watch carefully and press the squares in the same order".
4. Increase the trials in length from 2 to 5 blocks. Give the children three attempts to move to the next level of difficulty.
5. End the test when a child failed to correctly recall two sets at a given difficulty level. The examiner uses the mouse buttons to record whether the answer is correct. Calculate the scores based on the number of correct answers given.
Phoneme deletion
NOTE: This task included 15 two-syllable words: five with consonant-vowel (CV) first syllable structure, five with consonant-vowel-consonant (CVC) first syllable structure, and five with consonant-consonant-vowel (CCV) first syllable structure.
1. Say a word to the child and have them repeat it, omitting the first sound.
2. Have the robot say "In this game, you have to remove the first sound of each word. If you hear the word /tarde/ (late), you have to remove the sound /t/. So, you will say /arde/. Now, try it on your own".
3. The examiner uses the mouse buttons to record whether the answer is correct. Calculate the score based on the total number of correct responses.

Representative Results

In order to test the utility and effectiveness of this diagnostic tool, its psychometric properties were analyzed in a large-scale sample. A total of 933 Spanish primary school students (boys = 508, girls = 425; M_age= 10 years, SD = 1.36) from grade 2 to grade 6 (grade 2, N = 169 [89 boys]; grade 3, N = 170 [89 boys]; grade 4, N = 187 [106 boys]; grade 5, N = 203 [113 boys]; grade 6, N= 204 [110 boys]) participated in the study. The children were from intact classes at state and private schools in urban and suburban areas of Santa Cruz de Tenerife. Students were classified into two groups: a) MLD children with scores within or below the 16th percentile in a standardized arithmetic test (grade 2, N =14; grade 3, N =35; grade 4, N =11; grade 5, N = 47; grade 6, N = 42); and b) typically achieving children with scores within or above the 40th percentile in the same test (grade 2, N =130; grade 3, N =124; grade 4, N =149; grade 5, N = 110; grade 6, N = 105).

The multidimensionality of the tool's structure was tested by means of Confirmatory Factor Analysis (CFA) using the lavaan package in R⁶⁸. A five-factor model for BM-PROMA was hypothesized. A cognitive factor containing all domain-general tasks was expected, as the contribution of domain-general skills to mathematical performance is different from that of domain-specific skills⁶⁹^,⁷⁰. An arithmetic factor grouping only arithmetic tasks was also expected, as arithmetic and basic numerical skills involve different cognitive and brain correlates⁷¹ . Finally, following the Triple Code Model⁷², three factors grouping numerical tasks according to whether the task involves verbal, Arabic or analogic representations were expected.

Evidence concerning internal consistency was assessed using Cronbach's alpha. Cronbach's alphas were calculated for all measures and presented both for each grade and for the whole participant sample. Internal consistency values were considered excellent when α ≥ .80, good when α ≥ .70 and <.80, acceptable when α ≥ .60 and <.70, poor when α ≥ .50 and < .60, and unacceptable when α < .50⁷³.

Model goodness of fit was estimated using the robust maximum likelihood (RML) estimation method and assessed using the following indexes⁷⁴^,⁷⁵: standardized root mean square (SRMS ≤ .08), chi-square (χ², p> .05), Tucker-Lewis index (TLI ≥ .90), comparative fit index (CFI ≥ .90), root mean square error of approximation (RMSA ≤ .06), and Composite Reliability (ω ≥ .60). Modification indices (MI) were inspected.

Descriptive statistics were examined and are presented in Table 1. Results showed a normal distribution of the data, with kurtosis and skewness indexes lower than 10.00 and 3.00, respectively⁷⁶.

Measures	Grade 2		Grade 3		Grade 4		Grade 5		Grade 6		Total
Measures	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD
Missing numbers	3.81	3.29	5.79	3.49	7.68	3.15	7.56	3.50	8.33	2.98	6.67	3.65
Two-digit number comparison	2.02	.54	1.78	.35	1.50	.24	1.46	.15	1.44	.15	1.62	.38
Reading numbers	1.14	.27	1.27	.23	1.14	.21	1.17	.18	1.21	.20	1.24	.24
Place value	8.83	3.19	9.83	2.89	10.58	1.62	10.33	1.95	10.89	1.49	10.14	2.38
Number line 0-100	.11	.06	.07	.30	.06	.02	.05	.02	.05	.19	.07	.04
Number line 0-1000	.18	.09	.13	.06	.09	.04	.09	.04	.07	.02	.11	.06
Addition fact retrieval	5.11	4.42	7.03	5.24	11.15	5.74	10.27	5.82	12.03	5.30	9.32	5.93
Subtraction fact retrieval	4.36	3.79	5.78	4.66	8.94	4.53	8.64	4.84	9.76	4.31	7.66	4.89
Multiplication fact retrieval	2.92	3.27	6.32	4.97	11.48	5.67	10.10	5.90	11.49	5.43	8.72	6.13
Arithmetic principles	8.33	4.71	8.05	3.41	8.95	3.80	9.38	4.01	10.78	4.56	9.21	4.22
Counting Span	4.57	2.35	5.45	2.65	6.41	2.56	6.43	2.59	7.03	2.49	6.05	2.67
Visuospatial working memory	6.26	2.74	7.30	2.62	8.18	2.33	8.46	2.42	9.27	2.23	7.98	2.66
Phoneme deletion	9.34	4.78	10.96	4.60	12.64	2.83	12.62	2.92	12.61	3.37	11.73	3.94
Rapid automatized naming- Letters	1.37	.32	1.53	.31	1.72	.31	1.80	.35	1.87	.36	1.68	.38

Table 1: Descriptive statistics of BM-PROMA subtests per grade.

The internal consistency of each measure, except for numerical working memory, is presented in Table 2. Results indicated α of above .70 for the majority of the measures at each grade, suggesting good to excellent internal consistency for most of the tasks.

Measures	Grade 2	Grade 3	Grade 4	Grade 5	Grade 6	Total	ICL
Missing numbers	.841	.843	.807	.858	.801	.861	1
Two-digit number comparison	.891	.925	.916	.868	.866	.895	1
Reading numbers	.861	.830	.849	.892	.753	.855	1-2
Place value	.843	.864	.722	.686	.740	.809	1-3
Number line 0-100	.825	.748	.658	.547	.678	.801	1-4
Number line 0-1000	.806	.820	.763	.743	.729	.867	1-2
Addition fact retrieval	.852	.879	.885	.892	.856	.898	1
Subtraction fact retrieval	.826	.880	.846	.868	.823	.876	1
Multiplication fact retrieval	.811	.861	.867	.881	.853	.901	1
Arithmetic principles	.586	.734	.844	.742	.866	.821	1-4
Visuospatial working memory	.741	.726	.660	.695	.699	.747	1-3
Phoneme deletion	.918	.933	.835	.853	.899	.911	1
Note. ICL = internal consistency level; 1 = excellent; 2 = good; 3 = acceptable; 4 = poor, 5 = unacceptable.

Table 2: Cronbach's a coefficient for all the measures at each grade.

In order to confirm the factorial structure of BM-PROMA, a CFA was conducted using the RML estimation method. The fit indices suggested an adequate fit of the five-factor model proposed for the data: χ² = 29.930 df = 67, p = .000; CFI = .948; TLI = .930; RMSEA = .053, 90% CI = [.046-.061]; SRMR = .046; F1, ω = .50; F2, ω = .75; F3, ω = .80; F4, ω = .81; F5, ω = .46 (Figure 11).

Figure 11: Confirmatory Factor Analysis of the BM-PROMA. Note. F1 = Arabic numerical representation factor; F2 = analogical representation factor; F3 = verbal representation factor; F4 = arithmetic factor; F5 =cognitive factor; RAN-L = rapid automatized naming- letters; VWM = visuospatial working memory; CS = counting span; PD = phoneme deletion; AP = arithmetic principles; MFR= multiplication fact retrieval; AFR = addition fact retrieval; SFR = subtraction fact retrieval; TNC = two-digit number comparison; RN = reading numbers; NL-100 = number line 0-100; NL-1000 = number line 0-1000; PV = place value; MN = missing number. Please click here to view a larger version of this figure.

The multidimensional approach of the tool was confirmed. The tasks included in BM-PROMA loaded on five factors: 1) the missing number and place value tasks loaded on the "Arabic Numerical Representation Factor"; 2) the number line estimation 0-100 and number line estimation 0-1000 tasks loaded on the "Analogical Representation Factor"; 3) the two-digit number comparison and reading number tasks loaded on the "Verbal Representation Factor"; 4) the arithmetic principles, addition fact retrieval, multiplication fact retrieval, and subtraction fact retrieval tasks loaded on the "Arithmetic Factor"; and 5) the counting span, phoneme deletion, RAN-L, and visuospatial working memory tasks loaded on the "Cognitive Factor".

In order to examine measurement invariance across grades, we split the sample into two groups. The first group was composed of students from grades 2-3 (Group A). The second group was composed of students from grades 4-6 (Group B). Students were regrouped to increase sample size and minimize the number of groups, as sample characteristics, the number of groups compared, and model complexity all affect measurement invariance⁷⁷. Four nested models were compared: configural (equivalence of model form), metric (equivalence of factor loading), scalar (equivalence of item intercept), and strict (equivalence of item residual). Results are presented in Table 3, which shows configural, metric, scalar, and strict invariance across groups.

Model	χ2	df	CFI	TLI	RMSEA (90% CI)	SRMR	ΔCFI	ΔRMSEA
Configural (structure)	364.145	134	.940	.918	.061 [.053 – . 068]	.051
Metric (loadings)	383.400	143	.937	.920	.060 [.053 – .067]	.056	- .003	-.001
Scalar (intercepts)	383.845	152	.939	.927	.057 [.050 – .064]	.056	.002	-.003
Strict (residuals)	398.514	166	.939	.933	.055 [.048 – .062]	.056	.000	-.002
Note. CFI = comparative fit index; TLI = tucker-lewis index, RMSEA = root mean square error of approximation;
CI = confidence interval; SRMR = standardized root mean square residual; Δ = difference.
All χ2 values are significant at p < 0.001.

Table 3: Fit indices for measurement Invariance of BM-PROMA.

Finally, Receiver Operating Characteristic (ROC) analysis was performed to study the diagnostic accuracy of BM-PROMA based on the five factors derived from the CFA analysis. The standardized Prueba de Cálculo Numérico (Arithmetic Computation Test)⁷⁸ was used as the gold standard for testing the accuracy of each single diagnostic measure (i.e., factors). Area Under the ROC Curve (AUC > .70), sensitivity (>.70) and specificity (> .80) values were explored⁷⁹. Results revealed acceptable AUCs for all factors in all grades except for F3 (i.e., verbal representation factor) in grades 3, 5 and 6, and F2 (i.e., analogical representation factor) in grade 2 (Table 4). Sensitivity and specificity values were highly variable, ranging from .468 to .846 for sensitivity and from .595 to .929 for specificity. These results denote that although all measures contribute to the development of mathematical competency, their utility varies across grades.

Grade	Factors	AUC	Sn	Sp
Grade 2	F1	.912	.808	.857
	F2	.902	.785	.929
	F3	.746	.823	.786
	F4	.906	.846	.929
	F5	.918	.838	.929
Grade 3	F1	.762	.734	.714
	F2	.736	.645	.800
	F3	.608	.468	.743
	F4	.753	.605	.771
	F5	.733	.556	.743
Grade 4	F1	.719	.745	.727
	F2	.694	.597	.727
	F3	.817	.705	.818
	F4	.775	.691	.818
	F5	.782	.678	.727
Grade 5	F1	.855	.764	.809
	F2	.810	.736	.745
	F3	.630	.527	.681
	F4	.835	.745	.809
	F5	.832	.855	.787
Grade 6	F1	.839	.686	.714
	F2	.776	.648	.738
	F3	.524	.486	.595
	F4	.891	.848	.905
	F5	.817	.752	.738

Table 4: Diagnosis accuracy of BM-PROMA subtests per grade. Note. F1 = Arabic numerical representation factor; F2 = analogical representation factor; F3 = verbal representation factor ; F4 = arithmetic factor ; F5 = cognitive factor; AUC = area under the curve; Sn = sensitivity; Sp = specificity.

Figure 1: Missing number task Please click here to view a larger version of this figure.

Figure 2: Two-digit number comparison task Please click here to view a larger version of this figure.

Figure 3: Reading numbers task Please click here to view a larger version of this figure.

Figure 4: Place value task Please click here to view a larger version of this figure.

Figure 5: Number line estimation task Please click here to view a larger version of this figure.

Figure 6: Arithmetic fact retrieval task Please click here to view a larger version of this figure.

Figure 7: Arithmetic principles task Please click here to view a larger version of this figure.

Figure 8: Counting span task Please click here to view a larger version of this figure.

Figure 9: Rapid automatized naming – letter task (RAN-L) Please click here to view a larger version of this figure.

Figure 10: Visuospatial working memory task Please click here to view a larger version of this figure.

Discussion

Children with MLD are at risk not only of academic failure but also of psycho-emotional and health disorders⁸^,⁹ and, later on, of employment deprivation⁴^,⁵. Thus, it is crucial to diagnose MLD promptly in order to provide the educational support that these children need. However, diagnosing MLD is complex due to the multiple domain-specific and domain-general skill deficits that underlie the disorder²²^,²³. BM-PROMA is one of the few computerized tools that uses a multidimensional protocol to diagnose primary school children with MLD, and the first to be standardized for Spanish-speaking children.

The present study has proven that BM-PROMA is a valid and reliable instrument. Results from ROC analyses were promising, showing AUCs ranging from .72 to .92 across almost all factors and grades. This indicates acceptable to excellent discrimination⁷⁹. The weakest support was found for F3 in grades 3, 5 and 6, and F2 in grade 4 yielded AUC < .70. It is important to note that we used only one measure as the gold standard, and that it is focused on multi-digit calculation skills; as such, it is a very limited measure. A gold standard should reflect the content of the criterion measure under investigation⁸⁰, so we consider that the classification accuracy could be improved by the addition of other standardized state assessments in future studies.

Although BM-PROMA is a very comprehensive tool, it would be relevant for future versions to include other domain-specific skills that have been found to be impaired in MLD children, for example, non-symbolic comparison tasks in younger children⁸¹ and rational number manipulation or the solving of arithmetic word problems⁸²^,⁸³ in older children. It would also be essential to incorporate other domain-general skills that seem to be deficient in MLD, such as inhibitory control⁸⁴.

Despite the limitations described, BM-PROMA is one of the few pieces of software designed to identify children with dyscalculia, and the present study has proven that it is a valid and reliable instrument. The internal structure represents the tool's multidimensional evaluation approach. It provides a broad cognitive profile for the child, which is relevant not only for diagnosis but also for individualized instructional planning. Furthermore, its multimedia format is highly motivating for the children and, at the same time, makes the assessment procedure easier.

Disclosures

The authors have nothing to disclose.

Acknowledgements

We gratefully acknowledge the support of the Spanish government through its Plan Nacional I+D+i (R+D+i National Research Plan, Spanish Ministry of Economy and Competitiveness), project ref: PET2008_0225, with the second author as principal investigator; and CONICYT-Chile [FONDECYT REGULAR Nº 1191589], with the first author as principal investigator. We also thank the Unidad de Audiovisuales ULL team for their participation in the production of the video.

Materials

Multimedia Battery for Assessment of Cognitive and Basic Skills in Maths

Universidad de La Laguna

Pending assignment

BM-PROMA

References

Henik, A., Gliksman, Y., Kallai, A., Leibovich, T. Size Perception and the Foundation of Numerical Processing. Current Directions in Psychological Science. 26 (1), 45-51 (2017).
Henik, A., Rubinsten, O., Ashkenazi, S. The “where” and “what” in developmental dyscalculia. Clinical Neuropsychologist. 25 (6), 989-1008 (2011).
Ghisi, M., Bottesi, G., Re, A. M., Cerea, S., Mammarella, I. C. Socioemotional features and resilience in Italian university students with and without dyslexia. Frontiers in Psychology. 7, 1-9 (2016).
Parsons, S., Bynner, J. Numeracy and employment. Education + Training. 39 (2), 43-51 (1997).
Sideridis, G. D. International Approaches to Learning Disabilities: More Alike or More Different. Learning Disabilities Research & Practice. 22 (3), 210-215 (2007).
Duncan, G. J., et al. School Readiness and Later Achievement. Developmental Psychology. 43 (6), 1428-1446 (2007).
Wu, S. S., Barth, M., Amin, H., Malcarne, V., Menon, V. Math Anxiety in Second and Third Graders and Its Relation to Mathematics Achievement. Frontiers in Psychology. 3, 162 (2012).
Reyna, V. F., Brainerd, C. J. The importance of mathematics in health and human judgment: Numeracy, risk communication, and medical decision making. Learning and Individual Differences. 17 (2), 147-159 (2007).
Geary, D. C., Hoard, M. K., Nugent, L., Bailey, D. H. Mathematical cognition deficits in children with learning disabilities and persistent low achievement: A five-year prospective study. Journal of Educational Psychology. 104 (1), 206-223 (2012).
Kaufmann, L., et al. Dyscalculia from a developmental and differential perspective. Frontiers in Psychology. 4, 516 (2013).
Wong, T. T. Y., Chan, W. W. L. Identifying children with persistent low math achievement: The role of number-magnitude mapping and symbolic numerical processing. Learning and Instruction. 60, 29-40 (2019).
Haberstroh, S., Schulte-Körne, G. Diagnostik und Behandlung der Rechenstörung. Deutsches Arzteblatt International. 116 (7), 107-114 (2019).
Kaufmann, L., von Aster, M. The diagnosis and management of dyscalculia. Deutsches Ärzteblatt international. 109 (45), 767-777 (2012).
Murphy, M. M., Mazzocco, M. M., Hanich, L. B., Early, M. C. Children With Mathematics Learning Disability (MLD) Vary as a Function of the Cutoff Criterion Used to Define MLD. Journal of learning disabilities. 40 (5), 458-478 (2007).
Ramaa, S., Gowramma, I. P. A systematic procedure for identifying and classifying children with dyscalculia among primary school children in India. Dyslexia. 8 (2), 67-85 (2002).
Dirks, E., Spyer, G., Van Lieshout, E. C. D. M., De Sonneville, L. Prevalence of combined reading and arithmetic disabilities. Journal of Learning Disabilities. 41 (5), 460-473 (2008).
Mazzocco, M. M. M., Myers, G. F. Complexities in Identifying and Defining Mathematics Learning Disability in the Primary School-Age Years. Annals of dyslexia. (Md). 53, 218-253 (2003).
Barahmand, U. Arithmetic Disabilities: Training in Attention and Memory Enhances Artihmetic Ability. Research Journal of Biological Sciences. 3 (11), 1305-1312 (2008).
Reigosa-Crespo, V., et al. Basic numerical capacities and prevalence of developmental dyscalculia: The Havana survey. Developmental Psychology. 48 (1), 123-135 (2012).
Hein, J., Bzufka, M. W., Neumärker, K. J. The specific disorder of arithmetic skills. Prevalence studies in a rural and an urban population sample and their clinico-neuropsychological validation. European Child and Adolescent Psychiatry. 9, (2000).
Geary, D. C., Nicholas, A., Li, Y., Sun, J. Developmental change in the influence of domain-general abilities and domain-specific knowledge on mathematics achievement: An eight-year longitudinal study. Journal of Educational Psychology. 109 (5), 680-693 (2017).
Cowan, R., Powell, D. The contributions of domain-general and numerical factors to third-grade arithmetic skills and mathematical learning disability. Journal of Educational Psychology. 106 (1), 214-229 (2014).
Rubinsten, O., Henik, A. Developmental Dyscalculia: heterogeneity might not mean different mechanisms. Trends in Cognitive Sciences. 13 (2), 92-99 (2009).
Peake, C., Jiménez, J. E., Rodríguez, C. Data-driven heterogeneity in mathematical learning disabilities based on the triple code model. Research in Developmental Disabilities. 71, (2017).
Chan, W. W. L., Wong, T. T. Y. Subtypes of mathematical difficulties and their stability. Journal of Educational Psychology. 112 (3), 649-666 (2020).
Bartelet, D., Ansari, D., Vaessen, A., Blomert, L. Cognitive subtypes of mathematics learning difficulties in primary education. Research in Developmental Disabilities. 35 (3), 657-670 (2014).
Geary, D. C., Hamson, C. O., Hoard, M. K. Numerical and arithmetical cognition: a longitudinal study of process and concept deficits in children with learning disability. Journal of experimental child psychology. 77 (3), 236-263 (2000).
Landerl, K., Bevan, A., Butterworth, B. Developmental dyscalculia and basic numerical capacities: a study of 8-9-year-old students. Cognition. 93 (2), 99-125 (2004).
Moura, R., et al. Journal of Experimental Child Transcoding abilities in typical and atypical mathematics achievers : The role of working memory and procedural and lexical competencies. Journal of Experimental Child Psychology. 116 (3), 707-727 (2013).
De Smedt, B., Gilmore, C. K. Defective number module or impaired access? Numerical magnitude processing in first graders with mathematical difficulties. Journal of Experimental Child Psychology. 108 (2), 278-292 (2011).
Andersson, U., Östergren, R. Number magnitude processing and basic cognitive functions in children with mathematical learning disabilities. Learning and Individual Differences. 22 (6), 701-714 (2012).
Geary, D. C., Hoard, M. K., Nugent, L., Byrd-Craven, J. Development of Number Line Representations in Children With Mathematical Learning Disability. Developmental neuropsychology. , (2008).
van’t Noordende, J. E., van Hoogmoed, A. H., Schot, W. D., Kroesbergen, E. H. Number line estimation strategies in children with mathematical learning difficulties measured by eye tracking. Psychological Research. 80 (3), 368-378 (2016).
Chan, B. M., Ho, C. S. The cognitive profile of Chinese children with mathematics difficulties. Journal of Experimental Child Psychology. 107 (3), 260-279 (2010).
Geary, D. C., Hoard, M. K., Bailey, D. H. Fact Retrieval Deficits in Low Achieving Children and Children With Mathematical Learning Disability. Journal of Learning Disabilities. 45 (4), 291-307 (2012).
Clarke, B., Shinn, M., Shinn, M. R. A Preliminary Investigation Into the Identification and Development of Early Mathematics Curriculum-Based Measurement. Psychology Review. 33 (2), 234-248 (2004).
David, C. V. Working memory deficits in Math learning difficulties: A meta-analysis. British Journal of Developmental Disabilities. 58 (2), 67-84 (2012).
Peng, P., Fuchs, D. A Meta-Analysis of Working Memory Deficits in Children With Learning Difficulties: Is There a Difference Between Verbal Domain and Numerical Domain. Journal of Learning Disabilities. 49 (1), 3-20 (2016).
Peng, P., et al. Examining the mutual relations between language and mathematics: A meta-analysis. Psychological Bulletin. 146 (7), 595-634 (2020).
Andersson, U., Lyxell, B. Working memory deficit in children with mathematical difficulties: A general or specific deficit. Journal of Experimental Child Psychology. 96 (3), 197-228 (2007).
Guzmán, B., Rodríguez, C., Sepúlveda, F., Ferreira, R. A. Number Sense Abilities , Working Memory and RAN: A Longitudinal. Revista de Psicodidáctica. 24, 62-70 (2019).
Passolunghi, M. C., Cornoldi, C. Working memory failures in children with arithmetical difficulties. Child Neuropsychology. 14 (5), 387-400 (2008).
vander Sluis, S., vander Leij, A., de Jong, P. F. Working Memory in Dutch Children with Reading- and Arithmetic-Related LD. Journal of Learning Disabilities. 38 (3), 207-221 (2005).
Lefevre, J. A., et al. Pathways to Mathematics: Longitudinal Predictors of Performance. Child Development. 81 (6), 1753-1767 (2010).
Simmons, F. R., Singleton, C. Do weak phonological representations impact on arithmetic development? A review of research into arithmetic and dyslexia. Dyslexia. 14 (2), 77-94 (2008).
Kleemans, T., Segers, E., Verhoeven, L. Role of linguistic skills in fifth-grade mathematics. Journal of Experimental Child Psychology. 167, 404-413 (2018).
Hecht, S. A., Torgesen, J. K., Wagner, R. K., Rashotte, C. A. The relations between phonological processing abilities and emerging individual differences in mathematical computation skills: A longitudinal study from second to fifth grades. Journal of Experimental Child Psychology. 79 (2), 192-227 (2001).
García-Vidal, J., González-Manjón, D., García-Ortiz, B., Jiménez-Fernández, A. . Evamat: batería para la evaluación de la competencia matemática. , (2010).
Gregoire, J., Nöel, M. P., Van Nieuwenhoven, C. . TEDI-MATH. , (2005).
Navarro, J. I., et al. Estimación del aprendizaje matemático mediante la versión española del Test de Evaluación Matemática Temprana de Utrecht. European Journal of Education and Psychology. 2 (2), 131 (2009).
Cerda Etchepare, G., et al. Adaptación de la versión española del Test de Evaluación Matemática Temprana de Utrecht en Chile . Estudios pedagógicos. 38, 235-253 (2012).
Van De Rijt, B. A. M., Van Luit, J. E. H., Pennings, A. H. The construction of the Utrecht early mathematical competence scales. Educational and Psychological Measurement. 59 (2), 289-309 (1999).
Ginsburg, H., Baroody, A. . Test of early math ability. , (2007).
Butterworth, B. . Dyscalculia Screener. , (2003).
Beacham, N., Trott, C. Screening for Dyscalculia within HE. MSOR Connections. 5 (1), 1-4 (2005).
Karagiannakis, G., Noël, M. -. P. Mathematical Profile Test: A Preliminary Evaluation of an Online Assessment for Mathematics Skills of Children in Grades 1-6. Behavioral Sciences. 10 (8), 126 (2020).
Lee, E. K., et al. Development of the Computerized Mathematics Test in Korean Children and Adolescents. Journal of the Korean Academy of Child and Adolescent Psychiatry. 28 (3), 174-182 (2017).
Cangöz, B., Altun, A., Olkun, S., Kaçar, F. Computer based screening dyscalculia: Cognitive and neuropsychological correlates. Turkish Online Journal of Educational Technology. 12 (3), 33-38 (2013).
Zygouris, N. C., et al. Screening for disorders of mathematics via a web application. IEEE Global Engineering Education Conference, EDUCON. , 502-507 (2017).
Jiménez, J. E., Rodríguez, C. . Batería multimedia para la evaluación de habilidades cognitivas y básicas en matemáticas (BM-PROMA). , (2020).
Nuerk, H. -. C., Weger, U., Willmes, K. On the Perceptual Generality of the Unit-DecadeCompatibility Effect. Experimental Psychology (formerly “Zeitschrift für Experimentelle Psychologie”. 51 (1), 72-79 (2004).
Nuerk, H. -. C., Weger, U., Willmes, K. Decade breaks in the mental number line? Putting the tens and units back in different bins. Cognition. 82 (1), 25-33 (2001).
Booth, J. L., Siegler, R. S. Developmental and individual differences in pure numerical estimation. Developmental Psychology. 42 (1), 189-201 (2006).
Case, R., Kurland, D. M., Goldberg, J. Operational efficiency and the growth of short-term memory span. Journal of Experimental Child Psychology. 33 (3), 386-404 (1982).
Denckla, M. B., Rudel, R. Rapid “Automatized” Naming of Pictured Objects, Colors, Letters and Numbers by Normal Children. Cortex. 10 (2), 186-202 (1974).
Milner, B. Interhemispheric differences in the localization of psychological processes in man. British Medical Bulletin. 27, 272-277 (1971).
Rosseel, Y. lavaan: An R package for structural equation modeling. Journal of Statistical Software. 48 (2), 1-36 (2012).
Knops, A., Nuerk, H. -. C., Göbel, S. M. Domain-general factors influencing numerical and arithmetic processing. Journal of Numerical Cognition. 3 (2), 112-132 (2017).
Torresi, S. Review Interaction between domain-specific and domain-general abilities in math’s competence. Journal of Applied Cognitive Neuroscience. 1 (1), 43-51 (2020).
Arsalidou, M., Pawliw-Levac, M., Sadeghi, M., Pascual-Leone, J. Brain areas associated with numbers and calculations in children: Meta-analyses of fMRI studies. Developmental Cognitive Neuroscience. 30, 239-250 (2018).
Dehaene, S. Varieties of numerical abilities. Cognition. 44 (1-2), 1-42 (1992).
Streiner, D. L. Starting at the beginning: An introduction to coefficient alpha and internal consistency. Statistical Developments and Applications. 80 (1), 99-103 (2003).
Zainudin, A. Validating the measurement model CFA. A handbook on structural equation modeling. , 54-73 (2014).
Brown, T. A. . Confirmatory factor analysis for applied reaearch. (9), (2015).
Kline, R. B. . Principles and practice of structural equation modeling. , (2011).
Putnick, D. L., Bornstein, M. H. Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review. 41, 71-90 (2016).
Artiles, C., Jiménez, J. E. Prueba de Cáculo Artimético. Normativización de instrumentos para la detección e identificación de las necesidades educativas del alumnado con trastorno por déficit de atención con o sin hiperactividad (TDAH) o alumnado con dificultades específicas de aprendizaje (DEA). , 13-26 (2011).
Hosmer, D., Lemeshow, S., Rod, X. Sturdivant. Applied Logistic Regression. , (2013).
Smolkowski, K., Cummings, K. D. Evaluation of Diagnostic Systems: The Selection of Students at Risk of Academic Difficulties. Assessment for Effective Intervention. 41 (1), 41-54 (2015).
Piazza, M., et al. Developmental trajectory of number acuity reveals a severe impairment in developmental dyscalculia. Cognition. 116 (1), 33-41 (2010).
Van Hoof, J., Verschaffel, L., Ghesquière, P., Van Dooren, W. The natural number bias and its role in rational number understanding in children with dyscalculia. Delay or deficit. Research in Developmental Disabilities. 71, 181-190 (2017).
Swanson, H. L., Jerman, O., Zheng, X. Growth in Working Memory and Mathematical Problem Solving in Children at Risk and Not at Risk for Serious Math Difficulties. Journal of Educational Psychology. 100 (2), 343-379 (2008).
Kroesbergen, E., Van Luit, J. E. H., Van De Rijt, B. A. M. Young children at risk for math disabilities: Counting skills and executive functions. Journal of Psychoeducational Assessment. , (2009).

Play Video

PDF

DOI

DOWNLOAD MATERIALS LIST

Cite This Article

Rodríguez, C., Jiménez, J. E., de León, S. C., Marco, I. Multimedia Battery for Assessment of Cognitive and Basic Skills in Mathematics (BM-PROMA). J. Vis. Exp. (174), e62288, doi:10.3791/62288 (2021).

Automatically Generated