Presentation Special Issue: Cognitive Science A special issue in cognitive science and the implementation of a comprehensive reviewing system

Fernando Marmolejo-Ramos^*
University of Adelaide

^*Fernando Marmolejo-Ramos, School of Psychology, Faculty of Health Sciences, University of Adelaide, Australia.

Correspondence concerning this article should be addressed to: School of Psychology, Faculty of Health Sciences, University of Adelaide, Adelaide, South Australia, Australia, 5005. Email: fernando.marmolejoramos@adelaide.edu.au, web page: https://sites.google.com/site/fernandomarmolejoramos/

Introduction

Cognitive science can broadly be defined as the interdisciplinary study of the mental operations displayed by a cognizer during its interactions with the internal and the surrounding environment (Marmolejo-Ramos, 2008; von Eckardt, 2006). The combination of explanations and methodological approaches offered by fields such as Psychology, Artificial Intelligence, Neuroscience, Linguistics, Anthropology, and Philosophy is one of the distinctive characteristics of cognitive science (Miller, 2003; Nickerson, 2006). Although cognitive science is part of the research agenda in various countries, there is evidence suggesting that this is not the case in Latin-America (Marmolejo-Ramos, 2008).

This special issue brings together the work that researchers around the world are currently carrying out on diverse topics in cognitive science and presents it to the research community in Latin America. The purpose behind this special issue is to motivate researchers on the continent to continue the studies presented herein and, ideally through networking with international researchers to initiate a rigorous research agenda in specific topics in cognitive science. This special issue offers a qualitatively comprehensive reviewing system to qualitatively, and a quantitatively assess of the manuscripts submitted. The following sections consider these two aspects.

Firstly, the articles that compose the special issue will be briefly introduced so that the reader has a general idea of the topics and methodologies that characterize current cutting-edge research in cognitive science. Secondly, the specificities of the reviewing system used will be presented along with the results obtained for the current special issue. Finally, some suggestions for the potential implementation of a comprehensive reviewing system in the publication of high-quality research in cognitive science in Latin America will be discussed.

Current topics and methodologies in cognitive science research

The current areas being addressed by cognitive scientists include memory, attention, vision, emotions, language, neuroscience and mathematical psychology. A collection of papers published in this special issue presents empirical evidence in relation to specific topics in each of these areas.

It is well known that text comprehension requires cognitive processes such as inferences, simulation, and memory; there is research in relation to each of these processes. In relation to memory processes, it has been suggested that episodic (E) and semantic (S) memory need the operations of the working memory (WM) in order to momentarily maintain active pieces of incoming information so that they can be linked to E and S memory systems. Elosúa et al. present a body of empirical evidence that suggests that a specific training in the executive functions of WM enhances reading comprehension in primary school children.

In broad terms, attention can be defined as the activity of focusing on one stimulus (or set of stimuli) while ignoring another (or another set of stimuli). For instance, participants can be asked to perform two sequential tasks and provide differential responses to each of them. In this special issue, Ruiz Fernández et al. report a study aimed at determining if the duration of response execution influences the processing of task order. The results indicate that participants optimize response scheduling in the case of dual-task situations.

Humans explore the environment via perceptual and motor senses such as touch, hearing, taste, smell, and sight. In particular, vision is one of the systems that has received considerable investigation due to its biological complexity and great effect on low- and high-order cognitive processes. Engmann and Cousineau show that stimuli with three redundant attributes in the visual modality (e.g., colour, form, and direction of movement) accelerate object recognition times. This is the first study of this kind to test how redundancy effect is increased (i.e. faster recognition times) due to more than two attributes that belong to the same modality. One of the core functions of the visual system is the use of gaze for the localization of objects. In two experiments, Yamada and Kawabe demonstrate that other person's gaze can affect the localization of objects and that attention deviates towards the gazed object or location. This is the first time that it has been suggested that mis-localisation of target objects is not only affected by distracter objects, but also by gaze cueing.

One of the greatest developments in cognitive science research is the recognition of emotions as a necessary component in human cognition. The recently formulated embodied cognition theory, argues that in metaphorical contexts emotionally valenced concepts are associated with sensorimotor spatial representations. For example, the word "happy" is associated with the spatial location "up" and with body movements that match that physical direction. Ansorge and Bohner used an Implicit Association Task to demonstrate that positively laden concepts are semantically associated with "up" spatial locations, whereas negatively laden concepts are associated with "down" spatial locations. Most researchers in the embodied cognition framework argue that perceptual and motor systems are always activated during the processing of emotion concepts. However, other researchers argue that motor systems are not always activated during the processing of emotionally valenced stimuli and that perceptual systems can take sole control of the process. In a full-length article, Marmolejo-Ramos and Dunn report empirical evidence in this regard.

Research into language has been traditional in cognitive science for many years. However, most studies in psycholinguistics have focused on the English language in order to build theories and put forward claims concerning the interactions between language and cognition. In this issue, Mishra uses an eye-tracking method to demonstrate that children and adult speakers of Hindi (a free word order and verb final language) exhibit different shifts in visual attention during sentence generation. His results specifically indicate that while adults attend primarily to the verbs in a sentence, children attended mostly to the subjects and objects in a sentence. Various methods have been crafted to test the idea that language is embodied, given that its processing requires the activation of perceptual and motor brain systems. Milin and Zdravkovic examine a classical technique called semantic differential (SD) that measures commonly understood meanings that words carry. The authors propose a methodologically modified version of the SD technique to show how the abstract and concrete concepts have perceptual and motor properties that can be represented in the semantic space.

Behavioural studies are the trademark of the experimental approach used by the researchers mentioned so far. However, other researchers resort to brain imaging techniques to address their questions of interest. Thus, Kondo and Watanabe use a more recent technique in neuroscience called magnetoencephalography (MEG) to demonstrate that brain activity reflects a correlation between the subjective perception of brightness and luminance processing in the human visual system. Likewise, Tagai et al. use a method known as near infra-red spectroscopy (NIRS) to show how Japanese and non-Japanese participants exhibit different brain activity in relation to stimuli portraying politeness.

Finally, another approach requiring further examination for greater understanding of human behaviour is that of computational modelling. This method in turn relies on mathematical and statistical formulas in order to propose law-like rules that represent cognitive, perceptual, and motor processes and that can be verified quantitatively. Ultimately, the computational modelling of psychological processes via mathematical/statistical concepts and language is known as mathematical psychology. In this special issue, Gómez and Erber implement a computational model that tests a recent claim in social psychology regarding reciprocity in romantic desire. Their results not only provide a parsimonious explanation but a challenge to current research carried out in social psychology. Even the most well-designed behavioural or neuro-scientific experiment or the most elegant computational model can go astray if the results obtained are not analysed properly. Scientists working in cognitive science, and many other fields, are based on statistical methods in order to make sense of the data obtained to render them into reportable results. Thus, Wilcox et al. propose two new statistical methods for pair-wise comparisons of data sets that are small and that resemble data obtained via Likert scales.

A comprehensive reviewing system for the publication of cognitive science research in Latin-America

As it is a current practice in all scientific journals, a peer-review system was used for the assessment of the manuscripts submitted to this special issue. In the traditional peer-review system, the guest editor assigns two to three reviewers whose expertise

matches the topic of the paper. In some cases, the authors can also submit a list of suggested reviewers and even reviewers who are opposed (for example, journals such as the Journal of Experimental Psychology: General and PLoS ONE welcome a list of (non) suggested reviewers). For this special issue, authors were asked to submit a list of four suggested reviewers; two of them were randomly chosen and asked to be official reviewers of the paper. Also, following the traditional reviewing system, both the reviewers' and the authors' identities were masked in order to minimise bias in the reviews. Additionally, authors were requested not to reveal in any form their identity in the paper.

It is also traditional that reviewers receive a template with which to assess the paper and perform a qualitative assessment of different sections of it, e.g., its abstract, introduction, etc. A standardised reviewing template (SRT) that combines a qualitative and a quantitative assessment of the papers was designed for this special issue. The quantitative assessment required that specific sections of the paper were evaluated on a Likert scale ranging from 1 (low quality) to 7 (high quality). Thus, the reviewer had to provide a quantitative assessment that reflected the qualitative appraisal given to specific sections of the paper. The sections of the paper evaluated were: i) title, ii) abstract, iii) keywords, iv) introduction, v) method, vi) results, vii) discussion, viii) conclusions, ix) and references. Reviewers were also asked to provide comments and ratings for the paper's x) originality and xi) relevance, and to give an xii) overall qualitative and quantitative appraisal of the paper.

Furthermore, reviewers could add additional general or specific comments. For each section reviewers were given specific aspects to consider as part of the evaluation based on the latest APA guidelines. A final average was computed using the mean score obtained across all twelve sections being assessed in order to decide whether the paper would be rejected, accepted with major modifications, or accepted with minor modifications (this point will be discussed later). Figure 1 illustrates sections of the SRT used for the evaluation of papers submitted to this special issue.

The results of the first reviewing round were sent to the authors so that they could address the comments raised by the reviewers. Once the authors returned their amended articles, these were sent back to their reviewers for a second reviewing round. Reviewers sent a modified SRT (SRTm) in order to provide a final assessment of the manuscript being evaluated. The SRTm required the reviewer to provide a qualitative and quantitative assessment of the revised manuscript taking into account the completeness of the amendments undertaken, the overall improvement, readability, scientific contribution, reception by readers and overall likelihood of publication. The results of the evaluation determined whether an article would be published in its revised form, published, with amendments or not published (see Figure 2). Decisions for each category were based on the SRT parameters (see details in later sections).

Validation of the reviewing template used for this special issue

Thirty one researchers, including a number who were neither reviewers nor authors in this special issue, assessed the standardised reviewing template (SRT) based on the instruction in Figure 1.

The aspects evaluated were the reviewing template's presentation, informativeness, clarity, comprehensiveness, and usability. In addition, researchers were asked to provide an overall appraisal of the SRT. Researchers were also asked to provide qualitative and quantitative assessments of each of these components, and were given the option to provide specific and general comments on the template. Table 1 presents the results of the assessment on the reviewing template.

The grand mean rating indicates that, among all things considered, the template is a rather useful tool for the reviewing of articles. Indeed, a robust estimator of central tendency suggests an even more promising picture, i.e., the grand median and its deviation (MAD) were 6 and 1.48 respectively. Thus, it seems that the SRT can be considered as a "highly complete" tool to be used for the qualitative and quantitative assessment of scientific papers.

However, the researchers provided comments on certain aspects of the SRT that need to be revised and that, if addressed, will enhance its effectiveness. The following list presents potential solutions to the current drawbacks of the SRT:

• That certain aspects of the article that are not totally decisive should be merged under other categories. For instance, "keywords" and "abstract" can be merged into one category.
• That a robust estimator is used to compute a final average along with its 95% confidence intervals. The use of a robust estimator (e.g. the median) ensures an unbiased average and the confidence intervals (e.g. non-parametric bootstrap confidence intervals) can be instrumental in making an educated-decision as to the fate of the article.
• That reviewers are requested to use decimal places in the Likert scale. Working with decimal places leads to more accurate results and enhances statistical computation of averages and confidence intervals. Indeed, if the SRT is presented as an on-line form, it would be ideal that the rating be performed via a slider rating scale that uses decimal places and that is anchored by bipolar opposite labels.
• That the SRT is used as a document that can be completed on-line, that has spelling capabilities, that can be opened using commercial web browsers (e.g., Firefox, Google chrome, etc.), and that works in different operating systems (Macintosh, Windows, etc.). By doing so, word-processing or spreadsheet applications that might cause technical problems could be bypassed (a similar approach is already being implemented in the online and interactive reviewing system used by the Frontiers journals).
• lThat other aspects that account for articles other than empirical (e.g. methodological articles, narrative review articles, and quantitative review articles) be included. For instance, the aspect of "accuracy/completeness of statistical/ mathematical equations" could be added in the case of methodological articles; the "quality of tables and figures" could be used for empirical and review articles, and the "relevance to readership", "level of scholarship", "overall quality", "novelty", and "soundness" could be added for all types of articles.
• That some minor aspects of an article, e.g., "title" and "references", be down-weighted. However, it could be argued that two approaches might help in this case: i) the use of robust estimators to determine the final average rating and ii) providing more guiding and precise questions for seemingly minor aspects of an article. For instance, "references" might seem a minor aspect if the reviewer has to assess their APA appropriateness. However, if the reviewer is questioned as to the relevance of the literature used, the number of references consulted/cited, how recent they were, etc., the relevance of the "reference" could increase.

How decisions were made as to the rejection/acceptance of manuscripts

The SRT devised for this special issue relied on the quantitative averaged results of the ratings performed on various aspects of the manuscripts to determine their fate. For this purpose, the categories "reject", "accept with major modifications", and "accept with minor modifications" were conceived. In order to make the reviewing process clear and transparent, both authors and reviewers had access to the SRT form.

The average ratings and the three categories were paired with three quality levels as follows: 1 — 3: low quality [reject], 3.1 — 5: average quality [accept with major modifications], and 5.1 — 7: high quality [accept with minor modifications]¹. The ranges used for the SRT aimed at giving similar probabilities to an article of being rejected (R) (0.33), accepted with major (AM) and minor modifications (Am) (0.316 each). Overall, the probability of an article being accepted overweighted its chances of being rejected, 0.63 vs 0.33. The reason for this tendency towards acceptance is mainly based on the fact that experienced researchers were selectively invited to contribute to papers. Thus, it was assumed, much as priors are set in Bayesian statistics, that contributions would have a high chance of being of good quality.

The acceptance ranges of the present SRT can be altered to suit different acceptance/rejection probabilities. For instance, a range from 1-4 could represent R, 4.1-5.5 could represent AM, and 5.6-7 could represent Am. In this case, the probabilities of R would be 0.5, AM would be 0.233, and Am would be 0.233. Indeed, if some extra degree of strictness is sought in the case of acceptance, the span of the acceptance ranges could be further modified so that the range 4.1-5.9 represents AM and the range 6-7 represents Am. Under these new acceptances ranges, articles would have a 0.5 probability of being rejected, a 0.3 probability of being AM, and a 0.166 probability of being Am. Whatever the acceptance/ rejection rates are, it is essential that authors are informed of the reasons leading to the rejection of a paper (see Gernet, 2008).

Although most journals report on impact factor (IF), not all of them report the acceptance probability (usually called the "rejection rate", RR). Thus, the present SRT might assist in minimising the RR/ IF ratio of a journal by setting explicit rejection probabilities (as discussed above) in order to prevent cases in which journals with low IFs have very high RRs (see Aarssen et al., 2008). For example, a journal whose RR is 0.6 and IF is 1.283 would be preferable to one that has the same IF but an RR of 0.8 since chances of publication would be higher in the former while the IF of both journals is the same. Furthermore, Aarseen et al. (2008) suggest that journals should publish their RRs along with their IFs sinc they can also be an indicator of the quality of the papers published and, by default, the journal's quality. Note, however, that although relying exclusively on IFs has been a trend in the sciences for a long time, it has recently been demonstrated how misleading IFs can be (see Abin & Mabe, 2000; Mayor, 2010).

Overall quality of the articles published in the current special issue

One key advantage of the SRT proposed herein is that it requires that the qualitative and the quantitative assessments of the article are coherent. That is, a low qualitative assessment implies a low rating, whereas a high qualitative assessment requires a high rating. However, whereas the qualitative reports are prone to different interpretations, quantitative results lead to more precise conclusions. As each manuscript was evaluated by two reviewers, a two-sample permutation resample test (without replacement and using 10,000 simulations) was used to determine whether there was a significant difference between the mean ratings given by the two groups of reviewers in the first reviewing round (see chapter 3 in Chihara & Hesterberg, 2011 for a description of this type of tests).

The results showed that the observed difference between means (Or1 _g1-g2) was not significant, Or1 _{1- 2} = 0.36, p = 0.21 (see Figure 3A). These results thus indicate that there was an overall agreement between reviewers on the quality of the articles. Thus, averaging the ratings given by the two groups of reviewers to each manuscript suggested that all articles were of a rather high quality, M _{first d} = 5.25, 95% CI_b = [4.78, 5.63], SD, _d = round ' bca ^L'^J' first round 1.14, range _{f d} = 3-7, n = 28². The same profirst round cedure was performed on the results of the second reviewing round. The results showed that the observed difference between means (Or2 _g1-g2) was not significant, Or2 _{1- 2} = -0.34, p = 0.72 (see Figure 3B). As in the first reviewing round, these results suggest an overall agreement between reviewers on the quality of the amended articles. An average of the ratings given by all the reviewers indicated that articles were of a high quality, M _{second round} second round = 5.68, 95% CI_b = [4.93, 6.06], SD _{d d} = bca second round 1.18, range _{second round} = 2-7, n = 24. There was an second round improvement in the average rating of the articles' quality from round one to round two, i.e. from 5.25 to 5.68; however, the observed difference, O _{rr2 rr1} = 0.43, was not statistically significant as suggested by a permutation test, p = 0.1 (see Figure 3C).

It should be noted that the current analysis was intended to provide a comprehensive quantification of the quality of the manuscripts during the two reviewing rounds. One advantage of the reviewing template, however, is that it requires that the reviewers provide qualitative and quantitative assessments of various sections of a manuscript. Thus, a robust correlation between both reviewers' ratings of each of the articles' sections can be expected, in order to obtain levels of agreement between reviewers. The correlation value obtained and the associated p-value can be used as indicators of the level of agreement between reviewers on an article, and the significance of such agreement, respectively. Alternatively, a permutation test can be performed between the ratings given by two reviewers to a specific article in order to determine whether reviewers have significant differences between their average scores.

Discussion and conclusions

This special issue is aimed to motivate cutting-edge research in selected topics in cognitive science in Latin America and to promote the use of a comprehensive reviewing template (labelled herein as SRT and SRTm) to construct it. Marmolejo-Ramos (2008) has already provided recommendations as to how a rigorous research agenda in cognitive science could be carried out in Latin America. However, it is traditional that the research results be published in order to make them available for replication and extension; thus, ultimately, leading to an expansion of knowledge. A very important and decisive stepping stone in this broadening of knowledge is having access to unrestricted and sound research. The term 'unrestricted' refers explicitly to research made public via open source journals and the term 'sound' refers to research that is thoroughly and transparently peer-reviewed. Next section will consider ideas regarding the role of a comprehensive reviewing system for the publication of sound research.

The standardised reviewing system (SRT) used for the construction of this special issue can be instrumental in the publication of research that must undergo a comprehensive review process. Specifically, given that the SRT proposed herein requires both a qualitative and a quantitative assessment of specific aspects of a manuscript, the implementation of a rigorous, balanced, and educated review of manuscripts is required from reviewers.

The construction and expansion of scientific knowledge is determined by the quality of research that is produced within specific research fields and the quality of research that is cross communicated between different research fields. A way to secure high-quality research is by filtering the work that is being published; the current peer-review systems aim at doing so. However, traditional reviewing systems have serious drawbacks that prevent the publication of excellent work or allow the publication of faulty research. Perhaps one of the major reasons for this is the lack of transparency used in the peer-review system (see Wicherts, Kievit, Bakker, & Borsboom, 2012). Another cause could be the templates used for the manuscripts review.

The goal of a comprehensive reviewing system is, therefore, to regulate the construction and expansion of knowledge by filtering the research that is submitted for potential publication. As Figure 5 illustrates, a knowledge filtering system (□) should be in place in order to filter research across different research fields (RF) and within themselves (Ts). Such a knowledge filtering system (KFS) should be characterized by having a transparent reviewing process, by using peer-reviewed approaches (Wicherts et al., 2012), and implementing comprehensive reviewing templates. Different RFs (e.g. RF₁, RF₂, and RF₃) can contribute independently to the construction of theories and concepts (T) and the development of statistical and methodological approaches (M). A specific RF (RF_X) can, in turn, adopt Ts and Ms developed by other RFs, in order to build their own Ts and Ms.

The construction of Ts and Ms within a specific RF leads to the expansion (#) of scientific knowledge within that field. More importantly, knowledge expansion in a RF has repercussions in the scientific knowledge being developed in other (emerging and traditional) RFs. For instance, the expansion in RF_X can have effects on the Ts and Ms being constructed in RF_p and RFq.

The KFS, illustrated in Figure 5, constitutes a comprehensive reviewing system that enables the dissemination of high-quality research, and it is the task of the journals in any field to implement it. The Frontiers journals are, to the best of my knowledge, the only ones implementing one of the characteristics of this system, i.e. they use a transparent reviewing system. Most journals do not provide the specificities of the reviewing process to the author and he/she rarely knows who reviewed his/ her article; i.e. the reviewing process traditionally occurs "behind closed curtains" (Wicherts et al., 2012). However, in a transparent reviewing system this is not the case. Thus, it is suggested that an interactive and transparent reviewing system be put into practice.

For instance, in the first reviewing round, two to three reviewers, out of a list of nominated reviewers, could be randomly selected to review the paper. In the second reviewing round, both authors and reviewers could be put in contact to work together on the amendment of the article. Indeed, it would be ideal if the identities of both authors and reviewers were revealed from the moment a manuscript is submitted to a journal, so that transparency in the reviewing process is enhanced.

Two other desirable characteristics of a KFS are the use of a peer-reviewed peer review approach and the implementation of comprehensive reviewing templates. As originally proposed by Wicherts et al (2012), there should be put into practice a 'peer-reviewed peer review' approach in which the examinations of an article are further examined by another set of reviewers. Comprehensive reviewing templates should be characterised by using both qualitative and quantitative assessments of different sections of an article in order to produce a complete evaluation. The SRT and the SRTm presented herein aim at contributing in this direction. Indeed, it is conceivable that the SRT and the SRTm be combined in order to produce a single reviewing template that can be used throughout different evaluation rounds.

It is thus the intention of this editorial to promote the adoption of a KFS, similar to the one proposed herein, by Psychology (and other scientific) journals in Latin America. As there is no journal in Latin America dedicated to publishing research in cognitive science, it would be ideal to see that this special issue motivates its gestation and that the KFS is at its core.

Acknowledgments

Two researchers reviewed each manuscript. Since 12 papers were part of this issue, 24 researchers should be acknowledged for their role as reviewers. However, one researcher requested not to have his name disclosed. The following researchers participated as reviewers in this special issue (no order implied): Jazmín Cevasco, Helge Str0ms0, Iring Koch, Hartmut Leuthold, Gernot Horstmann, Sascha Topolinski, Mario Fific, Michael Zehetleitner, Yuki Yamada, Helene Kreysa, Kat-sumi Watanabe, Vassilis Pelekanos, Art Glenberg, Petar Milin, David Vinson, Pablo Gómez, Hiroyuki Tsubomi, Atsunori Ariga, Fuminoro Ono, Hanako Ikeda, Siva Tian, Lisa Lix, and Rogier Kievit. The following researchers participated as reviewers of two papers that were withdrawn from this special issue (no order implied): Helen Mitchell, Sergio Chaigneau, Julio Santiago, and Caitlin Fausey.

Finally, I would like to thank Robyn Groves for proofreading this editorial.

Pie de página

¹In the SRTm these ranges were: 1 — 3: not publish, 3.1 — 5: publish if and only if further recommendations are addressed, and 5.1 — 7: publish in the current form. Whenever a reviewer gave an average rating between 3.1 and 5 to an article, it was sent back again to the author to address the comments. Once the author returned the reviewed article, it was sent back to the reviewer. The reviewer, in turn, was required to use the SRTm for a third assessment of the manuscript.
²There were 14 manuscripts submitted upon invitation and two reviewers examined each of them. For the second reviewing round, two submissions were withdrawn after the results of the first reviewing round. Additionally, in some cases, some reviewers did not provide a quantitative assessment of the articles for the second reviewing round. Therefore, the ratings' sample size used in this analysis was larger than that one used in the subsequent analyses.

References

Aarseen, L. W., Tregenza, T., Budden, A. E., Lortie, C. J., Koricheva, J., & Leimu, R. (2008). Bang for your buck: rejection rates and impact factors in ecological journals. The Open Ecology Journal, 1, 14-19.         [ Links ]

Abin, M., & Mabe, M. (2000). Impact factors: use and abuse. Perspectives in Publishing, 1, 1-6.         [ Links ]

Chihara, L., & Hesterberg, T. (2011). Mathematical statistics with resampling and R. N. J.: Wiley.         [ Links ]

Gernert, D. (2008). How to reject any scientific manuscript. Journal of Scientific Exploration, 22 (2), 233-243.         [ Links ]

Marmolejo-Ramos, F. (2008). A call to arms: Time to do cognitive science in Latin-America. International Journal of Psychological Research, 1 (2), 41-52.         [ Links ]

Mayor, J. (2010). Are scientists nearsighted gamblers? The misleading nature of impact factors. Frontiers in Psychology, 1 (215), doi: 10.3389/fpsyg.2010.00215        [ Links ]

Miller, G. A. (2003). The cognitive revolution: a historical perspective. Trends in Cognitive Sciences, 7, 141-144.         [ Links ]

Nickerson, R. S. (2006). Cognitive science: experimental methods. In L. Nadel (Ed.), Encyclopaedia of cognitive science. Doi: 10.1002/0470018860        [ Links ]

Von Eckardt, B. (2006). Cognitive science: philosophical issues. In L. Nadel (Ed.), Encyclopaedia of cognitive science. Doi: 10.1002/0470018860        [ Links ]

Wicherts, J. M., Kievit, R. A., Bakker, M., & Borsboom, D. (2012). Letting the daylight in: Reviewing the reviewers and other ways to maximise transparency in science. Frontiers in Computational Neuroscience, 6 (20), Doi: 10.3389/fncom.2012.00020        [ Links ]