Can bilinguals rate their proficiency accurately in a language background questionnaire? A correlation between self-rated and objective proficiency measures Bilíngues conseguem avaliar com precisão a sua proficiência em um questionário de histórico da linguagem? Uma correlação entre medidas de proficiência autoavaliada e objetiva

Research with bilinguals apply different measures to assess proficiency, one of them being language background questionnaires, which include questions about individuals’ experience and self-rated proficiency. Studies suggest that bilinguals can report their proficiency consistently with objective measures (MARIAN et al., 2007; LUK et al., 2013; GERTKEN et al., 2014; BRANTMEIR et al., 2012). Within this context, the goal of the present study was to investigate the relationship between two distinct measures of proficiency. In order to do so, we correlated participants’ selfrated proficiency (N = 112) in the Language Experience and Proficiency Questionnaire (QuExPLi Questionário de Experiência e Proficiência Linguística) with their scores on the TOEFL ITP. We also carried out a simple linear regression between the mean scores of the two measures. Results show a significant weak correlation between participants’ self-rated proficiency and their scores on the TOEFL ITP, as well as a significant regression equation. Also, when the scores on individual skills were correlated with participants’ self-rated proficiency on those skills (N = 16), significant moderate to strong correlations were found. These results add to research that have found that bilinguals are able to report their proficiency accurately; however, more studies in different contexts are needed.


Introduction
Many studies have investigated linguistic and non-linguistic effects of a bilingual experience in individuals (see VALIAN, 2015;BIALYSTOK, 2017; and ANTONIOU, 2019 for reviews). Even though there is a great number of studies on the topic, researchers are far from reaching a consensus about the effects of bilingualism on cognitive processing. Research exploring different aspects of cognition (attention and inhibition, for example) in relation to the use of more than one language have yielded inconsistent results (ADESOPE et al., 2010;GREENBERG, 2013;BRUIN et al., 2014;ANTONIOU, 2019). That is, there is evidence suggesting that bilingualism entails varied cognitive benefits (COSTA et al., 2008;BIALYSTOK et al., 2012), whereas other studies report null, mixed or negative effects of the daily use of two or more languages (PAAP; LIU, 2014;PAAP et al., 2015).
There are some potential ways of addressing the causes for such discrepancy, many of them having to do with issues related to methodological compatibility across Can bilinguals rate their proficiency accurately in a language background...
Revista da Anpoll, Florianópolis, v. 52, n. 1, p. 142-161, jan-maio, 2021 | 144 studies (LEIVADA et al., 2020). One possible explanation is that researchers vary in the way they select and classify their participants, using different criteria to determine who is bilingual and often not treating bilingualism as a spectrum (GROSJEAN, 2006;SURRAIN;LUK, 2017;LEIVADA et al., 2020). This, in turn, makes the comparison across studies and the replication of findings difficult. Another explanation might be related to the fact that, depending on the participant sample, conditions of testing, and cognitive measures, factors such as a high level of proficiency and/or education can be overshadowed by another group of factors such as bilingual trajectory or context of language use. (LEIVADA et al., 2020). Also, potential effects of bilingualism compete with other sources of adaptation, and for this reason, depending on the participant sample, monolinguals may have other advantages that conceal these effects (VALIAN, 2015;LEIVADA et al., 2020).
In order to further advance our comprehension of the influence a bilingual experience may have in general cognitive processes, it is of paramount importance that researchers have a good understanding of participants' characteristics and consider a range of experiences which can, along with bilingualism, affect an individual's cognition.
Given that bilingualism is a complex and continuous variable comprising multiple factors, it is therefore crucial that studies document and report participants' language experience in a more transparent manner (SURRAIN; LUK, 2017).
One widely used way of acquiring information about bilinguals' experiences in research is administering language background questionnaires. In addition to information about participants' experiences, these questionnaires also assess their proficiency in the languages they use. As it is known, proficiency is a variable that directly affects the performance of bilinguals in experimental studies (TREMBLAY, 2011). Here we aim to investigate to which extent bilinguals are able to accurately report their proficiency in self-assessed measures, comparing them to objective ones.

Scholl, Fontes & Finger
Revista da Anpoll, Florianópolis, v. 52, n. 1, p. 142-161, jan-maio, 2021 | 145 This construct can be measured in various forms, posing a challenge to research involving bilingualism because if proficiency is not assessed in a similar way across studies, the comparison of results and implications become more problematic. In order to have a good understanding of an individual's proficiency, the four skills (reading, writing, listening, and speaking) should be assessed in both of the bilingual's languages (ROMAINE, 1995).
Given that the ability to communicate in a language is a complex construct, one skill may not represent or assess bilinguals' proficiency sufficiently (POWERS, 2010). Therefore, the instruments used to assess language proficiency should take different aspects of the bilingual experience into consideration to allow for a more comprehensive measurement of the ability of an individual to communicate effectively using the languages they speak.
A proficiency measure which is used on a large scale worldwide is standardized proficiency tests. Language tests have been adopted as a condition for entry, permanence or obtaining citizenship in different nations as well as a way of assessment in the language classroom, and their focus is the ability of an individual to use a language (SHOHAMY, 2007). Standardized tests are used on a large scale due to the fact that they are convenient in terms of administration, scoring and result statements (KUNNAN, 2012). The central characteristic of these tests is the uniformity of testing and evaluation practices, including their development, administration, scoring, description and interpretation of results (KUNNAN; GRABOWSKI, 2013). Among the strengths of standardized tests is the fact that they allow comparisons between different populations, they have good psychometric properties, they can be administered in groups and their data is simple to analyze. On the other hand, these tests are expensive to be used with all research participants and they may not be appropriate for a given population (CHRISTENSEN et al., 2014). Moreover, it is often not possible to have the results of a standardized test in a short period of timebetween the participant selection and the data collection, for example (TREMBLAY, 2011).
Another possibility for determining proficiency is the use of self-assessment measures. These instruments -usually in the form of questionnaires -are frequently used in bilingualism research and also have their advantages and drawbacks. As for the advantages, they generally include the four skills, are fast and cheap to use in research (BAKER, 2011). Furthermore, they offer a more holistic measure of proficiency, in spite of being more subjective than standardized tests (LUK; BIALYSTOK, 2013). With Can bilinguals rate their proficiency accurately in a language background...
Revista da Anpoll, Florianópolis, v. 52, n. 1, p. 142-161, jan-maio, 2021 | 146 respect to the disadvantages of using these instruments, there is a possibility that participants evaluate their proficiency better or worse than it really is (GROSJEAN, 2006). Also, the participant's attitudes towards their languages and the status of these languages may affect the reliability of these instruments (ROMAINE, 1995).
Questionnaires are the most common instrument of self-assessed proficiency used in research. They are efficient in terms of time, effort and financial resources, being able to gather a large amount of information in a short time (DÖRNEY, 2003). In order to avoid misinterpretation of questions, especially with participants with a low level of proficiency, it is advisable to present the questionnaire in the participants' native language (WILSON, 1999).
Self-assessed proficiency measures are frequently used to predict participants 'linguistic performance on objective tests. Studies suggest that the relationship between self-rated proficiency and objective measures of linguistic performance varies according to the languages and the tasks used (MARIAN et al., 2007). There is evidence indicating that bilinguals can evaluate their language experience and proficiency in a manner corresponding to objective instruments (WILSON, 1999;LI et al., 2006;MARIAN et al., 2007;BIALYSTOK, 2013;GERTKEN et al., 2014). For example, Delgado et al.
(1999) compared bilingual students' (n = 80) self-rated proficiency (reading, writing, listening and speaking) with their results in four subtests of the Woodcock-Munõz Language Survey. Results suggest that the correspondence between self-rated proficiency scores and those obtained in objective tests depends on the languages and skills being evaluated. Even though the authors did not find correlations which are strong enough to justify the use of self-assessed proficiency measures alone, their findings showed that they can serve as a first instrument to be used in the participant selection phase of a project. Brantmeier et al. (2012) also found a positive correlation (r = 0.341; p < 0.01) between self-rated proficiency (reading, writing, listening and speaking) and an online language test (vocabulary, grammar, reading and writing) with Spanish advanced language students (n = 150). In another study, Gertken et al. (2014) compared participants' reported proficiency in a questionnaire with a standardized test (Oxford Placement Test) and found a strong positive correlation between the measures, indicating that participants can accurately rate their proficiency.

Scholl, Fontes & Finger
Revista da Anpoll, Florianópolis, v. 52, n. 1, p. 142-161, jan-maio, 2021 | 147 Notwithstanding, there is a lack of consistency in the methods researchers employ to measure language proficiency and the inadequate register of participants' proficiency makes comparisons among studies more difficult (TREMBLAY, 2011). For this reason, we believe that studies in the areas of bilingualism and second language acquisition can benefit from the use of standardized proficiency measures as well as from greater consistency in the description of participants and procedures in the studies. In addition, combining self-rated proficiency measures obtained in language background questionnaires with objective measures can provide a more comprehensive description of the samples in individual studies, which may therefore enrich our understanding of bilingualism as a life experience (SURRAIN; LUK, 2017).

The present study
The aim of the present study was to analyze the relationship between two distinct measures of language proficiency: self-rated proficiency and scores on a standardized test. More specifically, we investigated how accurately a group of adult Brazilian bilinguals and users of English as an additional language 1 report their proficiency in English in a language background questionnaire. To that end, we compared participants' self-rated proficiency with their scores on a standardized large-scale proficiency exam, the TOEFL ITP.
As a measure of self-rated proficiency, we used the Language Experience and Proficiency Questionnaire (QuExPLi -Questionário de Experiência e Proficiência Linguística), adapted from Scholl & Finger (2013). The instrument was developed to contribute to studies about bilingualism in Brazil by providing a language background questionnaire in Portuguese to be used with adult bilinguals with varied language backgrounds and proficiency levels. The focus of the questionnaire is to gather essential language experience information about participants for research involving bilingualism.
The questionnaire contains questions about participants' language background, that is, which languages they know, in which context and when they learned them as well as for how long they were immersed in those languages in different contexts. There are also questions about the current use of the languages, how often, in which contexts, and for which purposes they are used. To finish, there is a self-rated proficiency scale in four skills -reading, writing, listening and speaking -ranging from 1 (very poor) to 6 (proficient).
The objective measure used in this research was the Test of English as a Foreign Language (TOEFL), developed by the Educational Testing Service (ETS), which is the most well-known large-scale proficiency exam (KUNNAN, 2012). The test is taken by individuals who are not native speakers of English in order to be able to enroll in undergraduate or graduate courses at universities in English-speaking countries. The version of the test used in this study was the TOEFL ITP (Institutional Testing Program).
The TOEFL ITP is administered by the higher education institution where candidates are taking the test, usually for the purpose of placing students according to their level or monitoring their progress. The test comprises academic subjects and assesses candidates in listening comprehension (50 questions), structure and written expression (40 questions), and reading comprehension (50 questions). The maximum score in the test is 677, with scores from 337 to 459 being equivalent to the A2 level in the CEFR, 460 to 542 to the B1 level, 543 to 626 to the B2 level, and 627 to 677 to the C1 level (ETS).
Such correlation would also provide validity evidence to the QuExPLi. This would in turn allow us to contribute to research on bilingualism in Brazil, which is in need of a complete instrument to assess the use of Brazilian participants' language backgrounds and their proficiency levels.

Participants
Participants were 112 undergraduate and graduate students from the Federal University of Rio Grande do Sul (52 women, 60 men) who were taking the TOEFL ITP test provided by the University. The age of the participants ranged from 19 to 50 years old (M = 25.2; SD = 5.5). Participants were Brazilian native speakers of Portuguese, and most or all classes they attended at university were taught in Portuguese. English age of acquisition ranged from 5 to 21 years old (M = 10.3 years, SD = 3.4), confirming that all participants started learning English after having learned Portuguese. Most participants learned English at school, in language courses or on their own. Participants reported reading, using the Internet and watching television and movies as the main factors which contributed to their learning of the additional language. Information about participants' English language background can be found in Table 1. Speaking 0.6 0.9 *Scale from 0 (nothing) to 6 (very much) ** Scale from 1 (a few times a year) to 6 (every day) Source: own authorship.

Data was collected in different administrations of the TOEFL ITP exam in the Federal
University of Rio Grande do Sul. The questionnaires were completed on the day of the test while participants waited for the TOEFL exam to begin and handed in to the researchers before the test started. All participants signed a consent form (project number CAAE 37252514.5.0000.5347) prior to completing the questionnaire.

The QuExPLi questionnaire
The language background questionnaire used in this study was the Language Experience and Proficiency Questionnaire (QuExPLi -Questionário de Experiência e Proficiência Linguística, see Appendix), based on Scholl & Finger (2013). The questionnaire was adapted to be used with our sample, considering that the participants were users of English as an additional language who were taking a standardized English proficiency test.
Participants reported their English proficiency (reading, writing, listening and speaking) in the QuExPLi in a scale ranging from 1 to 6, being 1 = very poor and 6 = proficient.
The mean scores of self-rated proficiency are described in Table 2.

Data analysis
Participants' answers in the questionnaire and their scores in the TOEFL ITP were manually inserted into an Excel spreadsheet, which was exported to SPSS (Version 18.1). Pearson correlations were performed between participants' self-rated proficiency (reading, writing, listening, speaking, and the mean score of the skills) and their scores on the TOEFL ITP (total score). Pearson correlations were also performed between each skill individually (listening, writing, and reading) and their equivalent skill on the TOEFL ITP (listening comprehension, structure and written expression, and reading comprehension) for the participants for which we had access to the detailed scores report on the test (N = 16). A simple linear regression was used to further investigate the relationship between self-rated proficiency and the total scores on the TOEFL ITP.

Results
Self-rated proficiency in English (reading, writing, speaking and listening) was correlated with participants' scores in the TOEFL ITP. There was a significant positive correlation between the total score on the test and self-rated proficiency in listening (r = .227; p < .05). However, correlations with the remaining skills -reading (r = .131), writing (r = .169), and speaking (r = .155) -were not significant.
Mean scores in self-rated proficiency were also correlated with the total results in the TOEFL ITP. There was a significant positive correlation between the measures (r = .189; p < .05). The values of the correlations are shown in Table 3. A simple linear regression was carried out to further investigate the relationship between the mean self-rated proficiency and the total score on the TOEFL ITP. It was
From the 112 participants, we were able to access the scores of each skill separately on the TOEFL ITP of only 16 participants. Their performances were correlated on each of the three skills measured in the test with their self-rated proficiency. There was a strong significant correlation between self-rated listening proficiency and listening comprehension on the TOEFL ITP (r = .892, p < .001), and moderate significant correlations between self-rated writing and structure and written expression (r = .580, p < .05), and self-rated reading and reading comprehension on the TOEFL (r = .733, p < .001).

Discussion
The aim of the present study was to investigate how accurately a group of adult Brazilian bilinguals and users of English as an additional language report their proficiency in English in a language background questionnaire. In order to do so, we correlated participants' answers in the QuExPLi, adapted from Scholl and Finger (2013), with their scores in a standardized proficiency test, the TOEFL ITP. Taking previous research into consideration, (MARIAN et al., 2007;LI et al., 2006;LUK et al., 2013;GERTKEN et al., 2014;WILSON, 1999;BRANTMEIER et al., 2012) we expected to find a positive correlation between the two different measures of language proficiency. Our hypothesis was confirmed given that there was a significant correlation between the average score of the four skills (reading, writing, listening, and speaking) and the score on the TOEFL ITP.
There were strong correlations found when each skill was correlated separately with its equivalent in the test. In other words, the higher the participants rated their proficiency in the questionnaire, the higher their scores on the TOEFL ITP were. Nevertheless, when considering each self-rated skill individually, in relation to the total score on the exam, significance was reached only with listening (r = .227; p < .05).
Listening ability is required to a great extent in the test (50 questions would be expected that results should be similar in these two abilities. One explanation for these findings might be that participants are able to assess their listening skills better than the other ones due to a higher frequency of use of this skill, especially for leisure activities. When we observe the number of hours participants reported using the English language for different activities (Table 1) Even though we had access to the detailed score report of only a small number of participants -16 out of 112 -on the TOEFL ITP, when we correlated self-rated proficiency on listening, writing, and reading and the individual scores on listening comprehension, structure and written expression, and reading comprehension, significant positive correlations were found. In other words, the better participants rated themselves in these skills on the questionnaire, the higher their scores on these skills on the TOEFL ITP were. Self-rated listening was the skill that correlated most strongly with its counterpart on the objective test. Once again, this might be related to the fact that listening is the skill most used by the participants, making their self-evaluation more reliable.
The significant positive correlation of the average self-rated proficiency score in the questionnaire with the scores on the TOEFL ITP (r = .189; p < .05) is in accordance with previous studies (MARIAN et al., 2007;LI et al., 2006;LUK et al., 2013;GERTKEN et al., 2014;WILSON, 1999;BRANTMEIER et al., 2012) which found that bilinguals are able to report their language proficiency consistently with objective measures. Moreover, a simple linear regression also showed that self-rated proficiency (mean scores) significantly predicted the total scores on the TOEFL ITP, suggesting that the higher participants rated themselves, the higher they scored on the test.
These results suggest that self-assessment measures of proficiency are efficient and can be used when researchers select bilingual participants for studies. However, it should be noted that, even though the correlation between the two proficiency instruments in this study was significant, it is a weak one. One possibility for this correlation not being stronger is that the TOEFL ITP measures academic English skills, and when individuals report their proficiency in a language, they might think about it in a more general way, in situations they encounter in their everyday lives. In other words, participants' self-rated proficiency might have been thought more globally, while the proficiency measured by the TOEFL ITP is more specific to the academic context. A limitation of this study is that the objective measure of proficiency did not include speaking ability. In order to understand proficiency more globally, the four skills should be accounted for. Future research can investigate whether other standardized proficiency tests correlate with selfrated proficiency, in order to increase the knowledge about the association between these two types of measures.
In bilingualism and second language acquisition research, it is of great importance that researchers consider participants' language background and proficiency in order to gain a more comprehensive understanding of their bilingual experience. For this reason, having two proficiency measures, one more objective and the other more holistic and subjective, can be a great way to fully understand participants' proficiency and contemplate how it might relate to the variables being studied.
In conclusion, this study adds to research on bilingualism and second language acquisition that have found a correspondence between self-assessment and objective measures of proficiency. Given that associations were found, even though the objective measure used was a specific proficiency test that assessed academic skills, we believe more studies on the topic in different contexts should be encouraged. Until there is stronger evidence that participants can report their proficiency accurately in questionnaires, the use of a combination of self-assessed and objective measures is recommended for a better understanding of participants' proficiency.