SciELO - Scientific Electronic Library Online

 
vol.30 issue54Recruitment and Selection Devices in Financial Services Companies: Exploring organizational Re-production and legitimacy in Chile and the United Kingdom author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand

Article

Indicators

Related links

  • On index processCited by Google
  • Have no similar articlesSimilars in SciELO
  • On index processSimilars in Google

Share


Cuadernos de Administración

Print version ISSN 0120-3592

Cuad. Adm. vol.30 no.54 Bogotá Jan./June 2017

http://dx.doi.org/10.11144/javeriana.cao30-54.caef 

Artículos Producto de Investigación

Challenges in assessing the effectiveness of financial education programs: The Colombian case*

Retos en la evaluación de la efectividad de programas de educación financiera

Desafios na avaliação da efetividade de programas de educação financeira

Pilar Beatriz Álvarez-Franco**  , Melisa Muñoz-Murillo***  , Diego Alexander Restrepo-Tobón**** 

**Estudiante de Doctorado en Administración Estratégica de Empresas de la Pontificia Universidad Católica del Perú. Docente de tiempo completo en la Escuela de Economía y Finanzas de la Universidad EAFIT, Medellín, Colombia. Correo electrónico: palvare6@eafit.edu.co

***Estudiante de Magister en Finanzas en la Universidad EAFIT, Medellín, Colombia. Asistente de Investigación del Departamento de Finanzas, Escuela de Economía y Finanzas de la Universidad EAFIT, Medellín, Colombia. Correo electrónico: amunozm@eafit.edu.co

****Doctor en Economía y Finanzas de Binghamton University, Binghamton, USA, 2013. Docente de Tiempo Completo de la Escuela De Economía y Finanzas de la Universidad EAFIT, Medellín, Colombia. Correo electrónico: drestr16@eafit.edu.co

Abstract

Financial Education Programs -FEP- enjoy widespread governmental and private support and are considered essential tools for improving financial literacy, encouraging financial inclusion, and increasing consumer financial protection. Therefore, assessing their effectiveness is critical to guarantee that public and private resources are allocated wisely. The available empirical literature casts serious doubts on the effectiveness of FEP in achieving their objectives. Even properly designed -from an impact evaluation viewpoint- FEP fail to deliver long-run effects on individuals’ financial literacy or financial outcomes. We highlight the challenges to evaluate the impact of FEP. We showcase the international experience in assessing the effectiveness of these programs and offer a set of recommendations regarding the attributes that FEP should have to allow serious policy evaluation.

Key words: Financial literacy; financial education; impact evaluation

Resumen

Los Programas de Educación Financiera -PEF- gozan de apoyo público y privado generalizado, y son considerados herramientas esenciales en la promoción de la alfabetización y la inclusión financiera, y la protección del consumidor financiero. Por tanto, evaluar su efectividad es crítico para garantizar que los recursos son asignados eficientemente. La evidencia disponible alberga serias dudas sobre su efectividad. Incluso PEF diseñados adecuadamente -desde el punto de vista de la evaluación de impacto- fallan en generar efectos de largo plazo en la alfabetización y los comportamientos financieros de los individuos. Nosotros revisamos la experiencia internacional, destacamos los desafíos para evaluar el impacto de los PEF, y ofrecemos un conjunto de atributos que debe tener un PEF para permitir una evaluación de impacto seria.

Palabras-clave: alfabetización financiera; educación financiera; evaluación de impacto

Resumo

Os Programas de Educação Financeira -PEF- contam com apoio público e privado generalizado e são considerados ferramentas essenciais na promoção da alfabetização, da inclusão financeira e da proteção do consumidor financeiro. Portanto, avaliar sua efetividade é fundamental para garantir que os recursos sejam disponibilizados com eficiência. A evidência disponível alberga sérias dúvidas sobre sua efetividade. Inclusive os PEF desenhados adequadamente -do ponto de vista da avaliação de impacto- falham em gerar efeitos de longo prazo na alfabetização e no comportamento financeiro dos indivíduos. Revisamos a experiencia internacional, destacamos os desafios para avaliar o impacto dos PEF e oferecemos um conjunto de quesitos que um PEF deve ter para permitir uma avaliação de impacto séria.

Palavras-Chave: alfabetização financeira; valiação de impacto; educação financeira

Introduction

Recent research shows that financial illiteracy is widespread worldwide (Lusardi & Mitchell, 2011), which has put financial education and financial literacy programs in the national public policy agenda of most countries (Xu & Zia, 2012). Governments embracing Financial Education Programs (FEP) seem to believe that such programs are inescapably associated with better financial literacy and financial outcomes. Most FEP enjoy uncritical acceptance to the point that politicians, policymakers, and journalists use the terms financial education and financial literacy interchangeably. In their discourse, these terms are associated, and often confounded, with desirable financial outcomes. Governments worldwide embrace FEP as the panacea solution to increase financial literacy and to improve consumer financial outcomes.

In Colombia, about 81% of its citizens are unable to compute a simple interest rate, 72% save nothing, 23% cannot determine how much they spent the week before, only 20% report being able to face unexpected expenses, and only 59% report they have enough to cover expenses after retirement (Redy, Bruhn, & Tan, 2013). Moreover, in 2012, Colombian students performed the worst in the financial literacy component of the PISA test (Pisa, 2012). As a response, the national government is carrying out a national strategy to increase financial literacy and financial outcomes by including economic and financial education programs in the curricula of most primary and secondary schools. In addition, financial institutions are required by law to offer FEP to their current and potential consumers. For these reasons, considerable private and public resources are being dedicated to FEF and therefore, identifying the precise effects of such policies is necessary to guarantee that public resources are allocated wisely.

A necessary step to evaluate the effectiveness of FEP is to distinguish between financial education, financial literacy, and financial outcomes. Financial education refers to the process of providing individuals information, instruction or objective advice to improve their understanding of financial products, develop their skills to be aware of risk and opportunities, make informed choices and take effective actions for their financial wellbeing (OECD, 2005). Financial literacy, on the other hand, implies having financial knowledge, financial ability, or both (Huston, 2010). Financial outcomes refer to the skills, abilities, and behaviors regarding how people deal with financial matters (e.g. wealth accumulation, saving rates, acquiring insurance, or managing a bank account (Hastings, Madrian, & Skimmyhorn, 2013).

Despite the worldwide impetus gained by FEP; specially, after the U.S. Financial Crisis of 2008-2009, the empirical evidence regarding the relation between financial education, financial literacy, and financial outcomes, gives no clear indication that FEP are effective in increasing people's financial literacy and improving financial outcomes (Hastings, Madrian, & Skimmyhorn, 2013; Fernandes, Lynch, & Netemeyer, 2014; Miller, Reichelstein, Salas, & Zia, 2014; Kaiser & Menkhoff, 2016).

The main difficulty in establishing the effects attributable to FEP is the presence of multiple confounding factors like individual preferences, cognitive abilities, numeracy, simultaneous intervention programs, etc. (Fox, Bartholomae, & Lee, 2005). Most public FEP and policies are, however, poorly designed and evaluating their effectiveness is challenging and expensive1. To shed light on these issues, we reviewed the empirical literature related to the relationship between financial education, financial literacy, and financial outcomes. We found that there are no clear-cut results regarding these relations. The empirical evidence is inconclusive regarding the effects usually attributed to FEP on financial literacy and financial outcomes (Hasting, Madrian, & Skimmyhorn, 2013; Fernandes et al., 2014; Miller et al., 2015; Kaiser & Menkhoff, 2016).

To highlight the challenges in assessing FEP's effectiveness, we will showcase the Colombian experience. The Colombian case has several attractive features. First, by law, financial education is a right for Colombian consumers, and financial institutions have the obligation to promote and deliver FEP as instructed by the Financial Superintendence of Colombia (Law 1328 of 2009). In addition, Law 1450 of 2011 (National Development Plan 2010-2014), mandated the Ministry of National Education to define the set of basic financial and economic abilities that the Colombian curricula should include. Decree 457 of 2014 created a multiagency system to coordinate public and private financial education programs and related initiatives. Second, we will show that most of the FEP implemented in Colombia fail to include an evaluation component and rather they are assessed as an ongoing process of monitoring. To our knowledge, only two out of the three programs that do include an impact evaluation component find a positive and short-lasting impact of financial education on financial literacy, but none on financial outcomes. Based on the empirical literature regarding the design and evaluation of FEP, we will then lay out the key characteristics that FEP should have in order to increase the odd of being effective and a set of key attributes that their evaluation component should include to assess their effectiveness.

The paper is organized as follows. Section 1 presents the empirical literature regarding the relation between financial education, financial literacy, and financial outcomes. Section 2 highlights the main attributes that FEP should have to establish their effectiveness in improving financial literacy and financial outcomes. Section 3 evaluates the Colombian experience in assessing FEP. Finally, the findings, conclusion, and suggestions for future research are presented.

1. Literature review

Financial education is important because it allows a person to thrive and develop other capabilities. In that sense, financial literacy and financial education are potentially important in providing the basic knowledge and skills necessary to efficiently manage a person's financial resources and achieve adequate financial security throughout his life cycle (Hastings, Madrian, & Skimmyhorn, 2013).

Developed and emerging countries recognize the importance to financial literacy as a mean to increase financial inclusion and empower people to make better financial decisions (OECD, 2014). Consequently, many countries around the world have undertaken important steps to promote FEP (National Strategy for financial education). In 2015, 60 countries are designing, implementing or revising a national financial education strategy2 (OECD/INFE, 2016). In some countries, the national strategy for financial education is an addition to a wider framework aimed at enhancing financial inclusion and consumer protection (OECD/INFE, 2012).

FEP aim at improving financial literacy and changing consumer financial behaviors (Jump$tart Coalition, 2004). Nevertheless, many studies have documented the importance of external factors to explain the variation in financial literacy that goes beyond the scope of FEP. Johnson and Sherraden (2007) argue that young people's access to resources and institutions may affect their ability to apply the knowledge and skills learned in FEP. Minority and low-income youth are less likely to access the mainstream financial system (Hogarth, Beverly, & Hilgert, 2003), families with low credit scores are charged high-interest rates (Johnson & Sherraden, 2007), and people who had an allowance or investment when they were children are prone to better understand the importance of saving as adults (Kotlikoff & Bernheim, 2001).

In the same line, the empirical evidence in favor of a positive causal effect of financial education on either financial literacy or financial outcomes is limited and mixed (Hastings, Madrian, & Skimmyhorn, 2013). Hogarth, Beverly, and Hilgert (2003) find a high correlation between financial literacy and the probability of assuming desirable financial outcomes (e.g. paying bills on time, monitoring expenses, budgeting, paying the full balance of credit cards on time, saving a proportion of their salary, maintaining a cash cushion for emergencies, diversifying investments, and imposing financial goals). On the other hand, some empirical studies find almost no effects of FEP on financial outcomes (Jump$tart Coalition, 2006; Mandell, 2008).

Other studies have investigated the relation between financial education and financial outcomes. Bernheim, Garrett, and Maki (2001) exploit the change in financial education mandates in the United States as an exogenous variation of financial education to evaluate the long-term effects of financial education on the self-reported rate of savings and wealth accumulation. They find a significant effect of financial education on both. However, Cole, Sampson, and Zia (2011) using the same natural experiment to determine whether there is a causal relation between financial education and saving decisions, find that financial education has no effect on financial outcomes while cognitive ability significantly improves saving outcomes. Thus, the link between financial education and financial outcomes is still unclear3.

Lusardi and Mitchell (2005) and Hung, Meijer, Mihaly, and Yoong (2009) examine the link between financial literacy and retirement planning for the United States and find a high positive correlation between them. In contrast, Hung, Meijer, Mihaly, and Yoong (2009) examine the relation between financial literacy and other financial outcomes (e.g., retirement savings and retirement portfolios management) and find no association between them. Cole, Sampson, and Zia (2011) use two large household surveys for India and Indonesia. They find that financial literacy is positively correlated with having a bank account, even after controlling for per capita expenditure levels, household discount rates, and risk aversion. Nonetheless, in this study, expenditure levels, and not financial literacy, is a strong predictor of bank account use.

Regarding the central question of whether FEP influence financial behavior, recent meta-analysis studies seem to indicate that the effects of FEP on financial behavior are scant. Miller et al. (2015), Fernandes et al. (2014) and Kaiser and Menkhoff (2016) present evidence based on a meta-analysis of more than 115 financial education interventions in both developed and developing countries including randomized control trials (RCTs) and natural experiments. Fernandes et al. (2014) find overall unreliable effects of FEP on financial behavior. In contrast, Miller et al. (2015) show that FEP can be effective if directed to influence specific financial behaviors. More recently, Kaiser and Menkhoff (2016) find that the unconditional effectiveness of FEP on financial literacy and, particularly, on financial behavior is small.

Kaiser and Menkhoff (2016) broadened the number and type of studies investigated in Fernandes et al. (2014) and in Miller et al. (2015) and compared more than 429 effects reported in this literature. They compute the standardized mean difference (Hedges' g) in outcomes between treatment and control groups. The Hedges' g gives an indication of the size and direction of an effect in scale-free units that allows comparing effect sizes across studies. The average Hedges' g from all the reported outcomes analyzed by Kaiser and Menkhoff (2016) did not reach 0.2, a level categorized by Cohen (1977) as small4.

For the above reasons, the assessment of FEP becomes relevant and imperative under such a complex landscape. The evaluation of FEP should aim at establishing their effectiveness to modify financial outcomes and behaviors (Lusardi & Mitchell, 2014; Lyons & Neelakantan, 2008), yet few studies undertake either serious evaluations of the impact of FEP or careful cost-benefit analysis. Such studies are required to discern the merits of FEP and to make effective policy prescriptions (Lusardi & Mitchell, 2014; Fox, Bartholomae, & Lee, 2005).

Hastings, Madrian, and Skimmyhorn (2013) offer a critical review of the effectiveness of FEP. They argue that most studies in the literature show a positive correlation between financial literacy and financial behaviors and outcomes. However, this association cannot be taken as evidence that FEP should be an effective mechanism to improve financial outcomes. First, self-selection into FEP makes it difficult to identify the real effects, if any, of these programs on financial literacy or financial outcomes. Individuals engaging in FEP may possess unobserved characteristics that correlate with financial literacy and financial outcomes (Meier & Sprenger, 2007). Hung and Yoong (2013) find that individuals engaging in retirement financial advice programs are wealthier and have higher levels of financial literacy -measured and self-reported.

Second, unobserved factors can make some individuals more likely to engage in FEP and, simultaneously, lead to better financial outcomes. The empirical literature shows a relationship between cognitive abilities and financial outcomes (Hastings, Madrian, & Skimmyhorn, 2013). This problem may lead to self-selection problems if individuals with higher cognitive abilities are more likely to participate in such programs or if financial outcomes strongly correlate with cognitive ability, as has been demonstrated in the literature (Banks & Oldfield, 2007; Gerardi, Goette, & Meier, 2010; Christelis, Jappelli, & Padula, 2010; Grinblatt, Keloharju, & Linnainmaa, 2009). Third, omitted variable problems can bias empirical results. Research on the determinants of financial literacy find that impatience (Meier & Sprenger, 2013), cognitive ability (Cole, Sampson, & Zia, 2011), Lusardi, Mitchell, & Curto, 2010), peer characteristics (Lusardi, Mitchell, & Curto, 2010), and risk aversion (Van Rooij, Lusardi, & Alessie, 2011) are strongly related to financial literacy. Thus, without a proper account for these hard to measure variables, the estimated effects of financial literacy on financial outcomes may be unreliable.

Fourth, the author's revew underscores that the causality between financial education and financial outcomes is difficult to pin down. Financial literacy necessarily mediates the hypothesized association between financial education and financial outcomes, but individuals cite personal experience as the main factor in determining their financial learning, giving close to no role to financial education (Hilgert & Hogarth, 2003). Therefore, reverse causality is a major concern in assessing the relation between financial literacy and financial outcomes5.

The shortage evidence regarding the relation between financial literacy and financial outcomes may be due to the endogeneity problems mentioned above. To address this issue, researchers resort to estimating methods based on instrumental variables. For instance, Van Rooij, Lusardi, and Alessie (2011) use the financial situation of relatives to instrument financial literacy for individuals. They find that financial literacy positively impacts wealth accumulation and stock market participation. Lusardi and Mitchell (2009) use high school financial education mandates in the United States as an instrument for financial literacy. They find that advanced financial literacy levels positively impact retirement planning. However, Hung, Meijer, Mihaly, and Yoong (2009), using the same strategy, but different methodology to measure financial literacy, find that the instrument used by Lusardi and Mitchell (2009) is only weakly related to financial literacy.

Another solution to the problems of estimating the relation between financial education, financial literacy, and financial outcomes is to use control field experiments. Two related experimental studies about business literacy training for female entrepreneurs come from Karlan and Valdivia (2011) and Calderon, Cunha, and De Giorgi (2013). The former randomly assign the clients of a microfinance institution to treatment and control groups. The training consisted of 22 weekly sessions; additionally, a baseline survey before the intervention and follow-ups one and two years later were conducted. The authors find an effect of the training on business knowledge and practices (e.g. reinvestment of profits, innovations, and increments on sales and revenues). Calderon, Cunha, and De Giorgi (2013) in a similar work for rural Mexico find a positive impact of the program on participants' profits.

Bernheim and Garrett (2003) use national surveys as an evaluation tool. They examine the effects of different FEP offered in the workplace and find that such programs increase savings for workers with low and moderate saving rates; while the effects are statistically insignificant for workers with high saving rates. For total wealth, the evidence is inconclusive. The authors' explanation for these results is that most employers offer these seminars and programs because employees have a low disposition to save, and since the survey has no details of each program, the authors cannot control for the reasons why employers offer them. This pre-existing difference between participants and nonparticipants may underestimate the effects.

An important study for Latin America is presented in Bruhn, de Souza Leao, Legovini, Marchetti, and Zia (2014). They show the results of a comprehensive financial education program for 20.000 Brazilian high school students. The program includes teacher and parent training sessions, didactic and innovative materials, and a relevant curriculum according to the population. The authors find that financial education in the school increases the probability of having a bank account. Because the follow-up survey was conducted immediately after the intervention, the results are indicative only of short-term effects.

As the amount of attention and resources spent on FEP increase, it is imperative to assure the efficiency and relevance of these programs and their long-term impact on financial behavior. In this paper, we aim at proposing a set of attributes that should be considered when designing and assessing a FEP.

2. Set of attributes to design and evaluate FEP

Design Attributes

FEP naturally differ in scope and goals. For instance, some may be targeting a specific population age group while others may be targeting different demographic groups. Some may aim atjust increasing awareness about financial products and financial opportunities while others may aim at changing individuals' attitudes or behaviors. No matter their differences, they should meet some minimum quality standards that provide the basis for evaluating their effectiveness. We propose that such programs meet the following set of minimum criteria or standards: (a) outcome-based, (b) competence-based, (c) developmentally-oriented, (d) context-dependent, (e) flexible, and (f) measurable.

Outcome-based: Since financial knowledge is not sufficient for delivering financially literate individuals, FEP should stress the need for developing desirable financial outcomes and strengthen participants' financial competencies. Programs just stressing awareness of financial products and financial opportunities are unsatisfactory and subject to manipulation by programs' sponsors. As Kaiser and Menkhoff (2016) show, financial education has a larger impact on financial literacy than on financial behavior. However, the effects of FEP depend on the type of financial outcome targeted (e.g., influencing borrowing is more difficult than influencing saving behaviors).

Competence-based: FEP should focus on developing a set of core financial competencies in the target population. The specific financial competencies to be developed should be a function of the specific goals, target population, and the institutional setting (context). Core financial competencies are made of three main components: (a) financial awareness, knowledge, and understanding; (b) confidence, motivation, and attitudes; and (c) skills and behaviors. FEP have a greater opportunity to be effective if they are designed to attack these three components while keeping the focus on some specific financial outcomes.

Developmentally-oriented: In addition to aiming at influencing outcomes and developing financial competencies, FEP should be designed to be relevant to the decisions that participants are currently facing. For instance, when targeting elementary school students, the contents and competencies should focus on what their immediate needs are and not on far into the future decisions. Likewise, youth participants aged between 10 and 15 may not necessarily be thinking about saving for retirement but may be very interested in having a working knowledge on how to start gaining financial independence. In other words, FEP should be concerned with understanding how children grow up to become financially capable adults. A good starting point could be the skills-based developmental model proposed by the U.K.'s Consumer Financial Protection Bureau (2016). According to this model, there are three building blocks for developing financially capable individuals. First, the executive function, which involves the mental processes we use to manage information and to control our behaviors such as prioritizing tasks, setting goals, and controlling impulses. Second, financial habits and norms that refer to the practices people use to manage their day-to-day financial lives including rule-of- thumbs in decision-making. And third, financial knowledge and decision-making skills that include factual knowledge, the ability to plan, set financial goals and conduct financial research. Individuals develop these building blocks from childhood to adulthood, and each is relevant only as a function of people's immediate concerns.

Context-dependent: FEP should be designed to account for the institutional environment relevant for participants. Compared to urban environments, rural areas lack financial access and participants may need differential access to financial products or may have different expectations regarding the usefulness of investing in financial literacy or capabilities. In addition, some aspects that are relevant for developed countries may not apply to developing or low-income countries. Thus, FEP should take these differences into account to tailor and develop their contents and intended competencies according to the participants they target.

Flexible and measurable: FEP should be designed in a way to accommodate the needs of participants without compromising their scope and goals. In addition, they should establish a meaningful and simple framework to assess the effectiveness of the programs that go beyond monitoring and head counting participants. As we will mention next, FEP should ideally have an impact evaluation showing evidence of their effectiveness. Without an effective assessment framework to measure programs' impacts, the objectives of these programs will be compromised.

Evaluation Attributes

FEP may be correctly designed and well-conceived according to the previous criteria, but their poor evaluation strategies might lead to wrong conclusions about their effectiveness. To be susceptible of impact evaluation, a FEP should have some minimum attributes or components. We propose that FEP should have the minimum following components: (a) valid counterfactual, (b) powerful sample selection, (c) instrument design, and (d) data collection.

Valid counterfactual: To remove confounding factors and establish causality, evaluators need to measure the outcome variable of individuals participating in the program and compare it with the outcome variable of these individuals without the program, which is obviously unobserved (Yoong, Mihaly, Bauhoff, Rabinovich, & Hung, 2013). The estimate of the latter is known as the counterfactual. Without a valid estimate of the counterfactual, an impact assessment is not possible.

There are two examples of invalid strategies to estimate a counterfactual. The first one compares participants with non-participants after intervention. The other one compares participants before and after intervention. Though highly used, these two methods yield unreliable counterfactuals (Duflo, Glennerster, & Kremer, 2007). In the first method, the difference between participant and non-participant outcomes may be generated by the pre-existing differences in their characteristics. In the second method, some variables that influence outcomes may change since the introduction of the program. So, neither of both methods allow the identification of a causal effect of the program.

Reliable methods to build a counterfactual are experiments and quasi-experiments (Yoong et al., 2013). Experiments randomly assign individuals in the sample to a treatment group -people who receive the program- and a control group -people who do not (Duflo, Glennerster, & Kremer, 2007). With a perfect randomization, treatment and control individuals are statistically similar in all dimensions, except in their participation in the program. Thus, the control group is a valid counterfactual. When randomization is not possible, quasi-experiments are a suitable option. They attempt to mimic randomness and to build a valid control group using econometric tools (Blundell & Dias, 2009).

The characteristics of a FEP determine the best method to build a counterfactual and not vice versa. For instance, when evaluations are retrospective, as they usually are, there is no possibility of treatment randomization and evaluators must use a quasi-experimental method. Yoong et al. (2013) provide a clear representation of the main criteria used to select the evaluation design (see Figure 1).

Source: Adapted from Yoong et al. (2013).

Figure 1 Decision tree for selecting impact evaluation design 

Powerful sample selection: A powerful sample size is the smallest sample that allows evaluators to capture true differences on outcome variables between treatment and control group individuals (Gertler, Martinez, Premand, Rawlings, & Vermeersch, 2011). A smaller sample leads to wrong conclusions about the effectiveness of the program, while a larger sample is expensive and requires more resources. Thus, the statistical power of a sample is always desired. To compute this powerful sample size, evaluators should compute power calculations6.

Instrument design: A good instrument should be comprehensible, relevant -to the target population, internally consistent, and replicable- for future surveys (Kempson, Collard, & Moore, 2006). Otherwise, the likelihood of obtaining responses that differ from their true values increases and the instrument will not measure what it is intended to. There are some international surveys already validated such as the financial capability surveys developed by FINRA and the World Bank, the financial literacy survey developed by OECD, and the high school and college student personal financial literacy surveys developed by the Jump$tart Coalition. Nevertheless, evaluators should carefully consider to what extent these instruments are relevant to their target population.

Data collection: It is also important to consider when and how often to collect data. Most high-quality evaluations conduct at least a baseline survey -before the intervention- and a follow-up survey -after the intervention (Gertler et al., 2011). How long after the intervention the follow-up survey should be conducted depends on whether FEP objectives are short-, middle-, or long-term ones. Some FEP may fail to find significant effects on behaviors because they do not include enough time to perceive changes in behaviors. Another concern is the drop-out rate from baseline to follow-up survey -known as attrition. High rates of attrition might yield differences between the treatment and the control group (Yoong et al., 2013), which invalidates the counterfactual and limits evaluators' ability to find causal relations.

3. The Colombian Case

Colombia has enacted laws and decrees to reinforce the regulatory environment to encourage financial inclusion (Giné, García & Gomez-Gonzalez, 2017). Law 1450 of 2011 (National Development Plan 2010-2014), mandated the Ministry of National Education to define a set of basic financial and economic abilities that the Colombian school curricula should include to promote economic and financial education to young people. Moreover, the Colombian government created a multiagency system to coordinate public and private financial education initiatives. The Decree 457 of 2014 established the Inter-Sectorial Commission for Financial Education responsible for proposing policy, guidelines, tools and methodologies for the adoption of the National Strategy for Economic and Financial Education. The Inter-Sectorial Commission will launch the national strategy official guiding document in June of 2017 and will conduct baseline and follow-up activities to review the existing level of financial education among the target audience and inform the evolution and coverage of the program.

On the other hand, the Financial Reform of 2009 (Law 1328 of 2009, Title I, Chapter III, Literal f) demanded financial institutions and financial industry associations to advocate for consumer financial protection and provide FEP to their clients. In response, different public and private institutions have launched FEP in the last six years. According to the Inventory Survey on Financial Education Programs/Initiatives in Colombia (Encuesta de Mapeo sobre Programas/Iniciativas de Educación Económica y Financiera en Colombia)7 of 2016, 113 institutions have at least a FEP, while 37 institutions were in the process of initiating one. Out of the 113 institutions with a program already implemented, 73 (65%) of them have developed the program within the previous six years.

The most frequently stated contents of these programs are (a) saving and borrowing attitudes, (b) savings, and (c) financial products' use; among others such as budgeting, credit access, attitudes towards consumption, insurance, consumer rights and obligations, and attitudes towards consumption. One interesting result is that banks focus on financial products' use (see Figure 2). 25 out of 33 banks' programs focus on financial products, which may signal that such programs can be being used as financial propaganda rather than as objective FEP. Conversely, financial products' use is not the first, but the fifth most frequently content stated by non-banking institutions (see Figure 3). Only 49, out of 80 non-banking FEP, focus on financial products.

Source: Authors elaboration using data from the Encuesta de Mapeo sobre Programas/ Iniciativas de Educación Económica y Financiera en Colombia (2016).

Figure 2 Contents of Colombian FEP offered by banking institutions 

Source: Authors elaboration using data from the Encuesta de Mapeo sobre Programas/ Iniciativas de Educación Económica y Financiera en Colombia (2016).

Figure 3 Contents of Colombian FEP offered by non-banking institutions 

Without proper evaluation of these programs, however, it is difficult to establish if financial institutions are using them to either benefit consumers or benefit themselves. This point is important to be considered by regulators since financial institutions' objectives may diverge from those in the public's interest. According to Hastings, Madrian, and Skimmyhorn (2013), sometimes firms have incentives to help naïve consumers, but sometimes they obtain benefits from consumer illiteracy (e.g. consumers who pay higher fees are likely to be less literate (Choi, Laibson, & Madrian, 2010; Giné, García, & Gomez-Gonzalez, 2017). Besides, the evidence is small regarding firms investing in informative advertising to offset consumer lack of financial knowledge (Hastings, Madrian, & Skimmyhorn, 2013). In informative advertising models, firms seek to reduce frictions and information costs. In persuasive advertising models, by contrast, firms seek to convince consumers about special characteristics of a product, generate brand loyalty, and reduce price sensitivity. Hastings, Hortagsu, & Syverson (2013) in a study for the private Mexican pension system find that firms tend to compete by persuasive rather than informative advertising to make workers less price-sensitive.

FEP with an evaluation component

To our knowledge, only three FEP in Colombia have conducted an impact evaluation study: Financefor change (Finanzas para el cambio), Promoting a saving culture (Promoción de Cultura del Ahorro), and Live safe (Viva Seguro). The first impact evaluation study comes from García (2012), who examines Finanzas para el Cambio. The program attempts to improve economic and financial knowledge, abilities, attitudes, skills, and behaviors of ninth- and tenth-grade students of low-income schools. As part of the program design, teachers of math, economics, and social science were trained each semester in the subjects mentioned above. 50 schools from Medellin, Cartagena, Cali and Bogotá participated in the program and the schools' principal self-selected their schools into the program.

This study had a total of 1.518 students, 781 assigned to a treatment group and 737 to a control group, eight treated and eight non-treated schools. The questions included were adapted from Lusardi and Mitchell (2005), the Jump$tart Coalition Financial Literacy 2009 survey, the Financial Education Evaluation Manual developed by the National Endowment for Financial Education (NEFE), plus some questions specifically designed for the program. For budgetary and practical reasons, the author selected the sample using the best convenience sampling method, a non-random method in which the criteria to choose the sample depends on the evaluator. Given the lack of randomness, this method generates a loss of external validity: the results cannot be generalized for all eligible population (Yoong et al., 2013). Thus, the 16 schools in the sample are not representative of all 50 schools participating in the program.

García (2012) builds an ex-post control group selecting schools that did not participate in the program. Given that treatment and control groups were not randomly built before the intervention and, therefore, this was a retrospective evaluation. Regardless of this solution, a selection-bias problem arose because the treated group had very specific characteristics (its members were more likely to work) compared to the control group. Propensity score matching, a quasi-experimental method, is then the most appropriate technique to construct a convincing counterfactual by matching individuals in the treated group to individuals in the control group based on the likelihood of participating in the program (Gertler et al., 2011). García (2012)'s overall results support significant effects of the program on the economic and financial knowledge of the participants. However, she finds no evidence in favor of positive effects on abilities, attitudes, skills, and behaviors.

The second impact evaluation study comes from Núñez et al. (2012), who examined Promoción de Cultura del Ahorro, a small program launched by the Colombian government in 2009 to improve financial access. The program has two components: financial education and monetary incentives. The educational component was developed during six workshops of two and a half hours each. The topics include budgeting, savings, debt management, and insurance. Participants receive monetary incentives in the form of quarterly raffles among mothers with an active account at the rural state bank Banco Agrario. The prize was ten times the average balance account of the last quarter. The authors randomize financial education and monetary incentives at the municipality level and conduct a baseline and a follow-up survey for 1.605 mothers in 2010 and in 2011, respectively. Three municipalities receive financial education, other three monetary incentives, three more receive both, and the last three, the control group, receive none. In the baseline, however, they find that treatments and controls differ. Two reasons explain the discrepancies. First, some municipalities started the program before the baseline survey. Second, the authors build clusters of municipalities according to population and poverty conditions, leading to a sample that does not represent all eligible municipalities.

To solve the endogeneity between the likelihood of participating and municipality poverty conditions, they use an instrumental variables methodology following a two-stage approach. In the first stage, they use a stereotype logistic regression, a multinomial model that yields consistent measures of the likelihood of belonging to one of the treatment groups: financial education, monetary incentives, or both, according to municipality characteristics. They argue that this regression solves the endogenous variation generated by the pre-existing differences between municipalities. The residuals from this regression capture the exogenous variation and can be used as a strong instrument. In the second stage, the authors use the residuals as the instrument to obtain the effects of participation on the outcomes by using a difference-in-difference methodology, which, in turn, solves the problems associated with the pre-existing differences at the individual level found in the baseline survey. Out of the 23 outcome variables they examine, they find increases in formal savings and saving capacities for all the treatment groups. However, they fail to find increases in access to formal financing and the use of financial products like debt and insurance.

Núñez et al. (2012) also use focal groups and surveys to collect data. The qualitative results show that the main obstacles households faced are in formal savings (the common use of purchase animals and cash accumulation as a way of saving, and the lack of knowledge about financial products). On the other hand, informal mechanisms of financing such as Christmas club accounts (natilleras or cadenas de ahorro in spanish)8 are preferred against formal financing because of the low transaction and search cost. Their overall results suggest an unbiased direction of the effect. Its magnitude, however, is difficult to rationalize. Thus, the results are uninformative about what is more effective: incentives, financial education, or both.

Rodríguez, Sanchez, and Zamora (2014) present the first impact evaluation of stream media Colombia. They examine Viva Seguro, a radio program on insurance that includes 36 daily sessions for 225 low-income station listeners. The content was didactic and relevant for the audience, comprising radio dramas, expert, and public interviews. To avoid dropouts, they made daily raffles of $100.000 and delivered a finaljackpot of $3.000.000 Colombian pesos. The main concern regarding this evaluation is its internal validity. Although there is randomness at the radio station level (the authors randomly assigned six radio stations from Barranquilla, Bogotá, and Pereira), the authors could not randomize at the individual level since listeners were loyal to their own radio station. Therefore, treatment and control individuals slightly differ in baseline: individuals in the treated group were more likely to be women. These differences persisted during the follow-up process given the high attrition rates: 80% in Barranquilla, 62% in Pereira, and 35% in Bogotá. Because of the high attrition rate in Barranquilla, the authors decided to remove this city from the analysis. The risk of attrition is that it contaminates the randomness set at the beginning of the intervention if people who drop out are different from people who do not. According to Yoong et al. (2013) most high-quality impact evaluations of financial capability programs seek to have an attrition rate of 5% or less.

Self-selection was another problem since listening to the program is not random: People who decided to listen to the program may be different from people who did not. To deal with the differences between treated and control groups, the authors use a difference-in-differences method. This method overcomes the problem that treated and control groups do not have the same pre-intervention conditions. However, it does assume that they have the same trends over time (Gertler et al., 2011). The authors also use Instrumental Variables as a robustness test given the potential endogeneity between the decision to participate and the outcomes. The instrument they use consists in whether the individual reported usually listening either to the treatment or to the control radio station.

Furthermore, Rodríguez, Sanchez, and Zamora (2014) examine the effects of the radio program on six outcomes: Number of risks and number of insurance products the individual knows about, knowledge on basic insurance concepts, attitudes towards insurance, perceived capabilities and knowledge on insurance, and changes in behavior. Their results hold amongst the two methodologies. They find impacts on knowledge of the number of risks and insurance products and perceived capabilities and knowledge on insurance. However, they fail to find a significant impact on participants' knowledge of basic insurance concepts, attitudes towards insurance, and changes in behavior.

Even though two of the three programs planned an impact evaluation before the intervention, the main difficulty faced by these authors is to control for the lack of randomness of their samples either in the baseline or in the follow-up survey. This affects mainly the internal validity of the results. In addition, high attrition rates are also detrimental for internal validity. The lack of information about financial literacy levels before the beginning of the program presents difficulties to overcome challenges to assess the real impact of the programs. Finally, budgetary constraints complicate the sampling and estimation process. Overall, the available evidence for Colombia fails to support a causal effect of FEP on either financial knowledge or financial outcomes, and there is no evidence on long-term effects.

Conclusion

Significant public and private resources are being used in developing FEP around the world. Colombia is riding this wave as well. Their objectives are broad: to increase individuals' financial literacy, to improve individuals' financial outcomes, to increase financial consumer protection, and promote financial inclusion. Recently enacted laws in Colombia mandate a general overhaul of primary and secondary education curricula to include the development of financial skills in the youth population. The law also mandates the financial industry to offer financial education programs to enhance financial literacy levels and improve financial consumer protection. How will the public know if such initiatives are worthwhile? We argue that without properly assessing the impact of such programs we take the risk of never knowing if they achieve their intended objectives. Thus, we may not know if the resources were wisely used.

In this paper, we showed that the international empirical literature offers little evidence regarding the effectiveness of financial education programs in either improving financial literacy or changing individuals' financial outcomes. We analyzed the Colombian experience regarding financial education and found that most programs lack a suitable impact evaluation component. Despite the large number of institutions -mostly financial institutions- carrying out FEP, most of them do not evaluate the results of those programs. Out of more than one hundred FEP and initiatives currently being developed, we identified just three programs for which a rigorous impact evaluation assessment was carried out. These studies reported short-term positive effects of the program on financial literacy levels but none on short- or long-term financial outcomes. Given the methodological challenges of these studies, the results should be taken with caution.

We also analyzed the publicly available information regarding these programs. Judging by the way the programs are being delivered, their content, and overall design, some of them seem to be ill-conceived and their intended impact cannot be assessed. Colombian regulators should carefully consider how to evaluate the impact of the current wave of FEP.

Finally, we recommend setting quality design and evaluation standards for FEP offered by either the government or private institutions. We propose that such programs meet the following set of minimum design criteria or standards: (a) outcome-based, (b) competence-based, (c) developmentally-oriented, (d) context-dependent, (e) flexible, and (f) measurable. In addition, to be susceptible of impact evaluation, a FEP should have the minimum following components: (a) valid counterfactual, (b) powerful sample selection, (c) instrument design, and (d) data collection.

A minimum set of requirements regarding the design and evaluation of the impact of such programs is a pre-requisite to guarantee that public resources are wisely allocated and that FEP serve the public interest. Without such a requirement, publicly endorsed FEP may distort the current Colombian educational curricula or serve only as financial propaganda for financial institutions.

References

Banks, J., & Oldfield, Z. (2007). Understanding pensions: Cognitive function, numerical ability and retirement saving. Fiscal Studies, 28(2), 143-170. [ Links ]

Bernheim, B. D., & Garrett, D. M. (2003). The effects of financial education in the workplace: Evidence from a survey of households. Journal of Public Economics, 87, 1487-1519. [ Links ]

Bernheim, B. D., Garrett, D. M., & Maki, D. M. (2001). Education and savings: The long-term effects of high school financial curriculum mandates. Journal of Public Economics, 80(3), 435-465. [ Links ]

Bruhn, M., de Souza Leáo, L., Legovini, A., Marchetti, R., & Zia, B. (2014). The impact of high school financial education: Experimental evidence from Brazil. World Bank Policy Research Working Paper, WP 6723. [ Links ]

Blundell, R., & Dias, M. C. (2009). Alternative approaches to evaluation in empirical microeconomics. Journal of Human Resources. 44(3), 565-640. [ Links ]

Calderon, G., Cunha, J. M., & De Giorgi, G. (2013). Business literacy and development: Evidence from a randomized controlled trial in rural Mexico. National Bureau of Economic Research, WP 19740. [ Links ]

Choi, J. J., Laibson, D., & Madrian, B. C. (2010). Why does the law of one price fail? An experiment on index mutual funds. Review of Financial Studies, 23(4), 1405-1432. [ Links ]

Christelis, D., Jappelli, T., & Padula, M. (2010). Cognitive abilities and portfolio choice. European Economic Review, 54(1), 18-38. [ Links ]

Christiansen, C., Joensen, J. S., & Rangvid, J. (2008). Are economists more likely to hold stocks? Review of Finance, 12(3), 465-496. [ Links ]

Cole, S., Sampson, T., & Zia, B. (2011). Prices or knowledge? What drives demand for financial services in emerging markets. Journal of Finance, 66(6), 1933-1967. [ Links ]

Consumer Financial Protection Bureau. (2016). Building blocks to help youth achieve financial capability: A new model and recommendations. Retrieved from: https://www.consumerfinance.gov/data-research/research-reports/building-blocks-help-youth-achieve-financial-capability/Links ]

Comisión Intersectorial de Educación Económica y Financiera. (2016). Encuesta de Mapeo sobre programas / Iniciativas de educación económica y financiera en Colombia. Tech. rep. [ Links ]

Departamento Nacional de Planeación. (2010). Plan Nacional de Desarrollo 2010-2014: Crecimiento Sostenibley Competitividad. Tech. rep. Retrieved from https://www.dnp.gov.co/Plan-Nacional-de-Desarrollo/PND-2010-2014/Paginas/Plan-Nacional-De-2010-2014.aspxLinks ]

Duflo, E., Glennerster, R., & Kremer, M. (2007). Using randomization in development economics research: A toolkit. Handbook of Development Economics, 4, 3895-3962. [ Links ]

Fernandes, D., Lynch, J., & Netemeyer, R. (2014). Financial Literacy, financial education, and downstream financial behaviors. Management Science, 60(8), 1861-1883. [ Links ]

Financiera, S., Financieras, I., & Cooperativas, E. (2010). Estrategia Nacional de Educación Económica y Financiera: Una propuesta para su implantación en Colombia. Retrieved from https://www.superfinanciera.gov.co/SFCant/ConsumidorFinanciero/estratenaledufinanciera012011.pdfLinks ]

Fox, J., Bartholomae, S., & Lee, J. (2005). Building the case for financial education. Journal of Consumer Affairs, 39(1), 195-214. [ Links ]

García, N. (2012). El impacto de la educación económica y financiera en los jóvenes: el caso de Finanzas para el Cambio. Borradores de Economía, 687, 2-47. [ Links ]

Gerardi, K., Goette, L., & Meier, S. (2010). Financial literacy and subprime mortgage delinquency: Evidence from a survey matched to administrative data. Federal Reserve Bank of Atlanta Working Paper Series 2010-10. [ Links ]

Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. (2011). Impact evaluation in practice. Washinton D.C: World Bank Publications. [ Links ]

Giné, X., García, N., & Gomez-Gonzalez, J. (2017). Financial information in Colombia. World Bank Policy Research Working Paper Series 7998. [ Links ]

Grifoni, A., & Messy, F.-A. (2012). Current status of national strategies for financial education. [ Links ]

Grinblatt, M., Keloharju, M., & Linnainmaa, J. (2009). Do smart investors outperform dumb investors? Chicago Booth Research Paper, 9(33), 46. [ Links ]

Hastings, J. S., & Mitchell, O. S. (2011). Howfinancial literacy and impatience shape retirement wealth and investment behaviors. National Bureau of Economic Research, WP 16740. [ Links ]

Hastings, J. S., Hortagsu, A., & Syverson, C. (2013). Advertising and competition in privatized social security: The case of Mexico.National Bureau of Economic Research, WP 18881. [ Links ]

Hastings, J. S., Madrian, B. C., & Skimmyhorn, W. L. (2013). Financial literacy, financial education, and economic outcomes. Annual Review of Economics, 5(1), 347-373. [ Links ]

Hilgert, M. a., & Hogarth, J. M. (2003). Household financial management: The connection between knowledge and behavior. Federal Reserve Bulletin, 106 (July), 309-322. [ Links ]

Hogarth, J., Beverly,S. G., & Hilgert, M.(February, 2003). Patterns of financial behaviors: Implications for community educators and policymakers. Federal Reserve System Community Affairs Research Conference. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.196.5257Links ]

Hung, A., & Yoong, J. (2013). The Market for Retirement Financial Advice. In 0. S. Mitchell, & K. Smetters (eds.). New York: Oxford University Press. [ Links ]

Hung, A., Meijer, E., Mihaly, K., & Yoong, J. (2009). Building Up, spending down: Financial literacy, retirement savings management, and decumulation. RAND Working Paper Series WR- 712. Retrieved from http://www.ssrn.com/abstract=1520203Links ]

Huston, S. J. (2010). Measuring financial Literacy. Journal of Consumer Affairs, 44(2), 296-316. [ Links ]

Jump$tart Coalition (2004, March). Jump$tart sees surge in legislation promoting financial literacy. Jump$tart Press Release. Washington D.C. Retrieved from http://www.jumpstart.org/fileuptemp/ACF17B5.docLinks ]

Jump$tart Coalition (2006b, April 5). Financial literacy shows slight improvement among nation's high school students. Jump$tart Press Release. Retrieved from http://www.jumpstartorg/fileuptemp/2006GeneralReleaseFinal%202.docLinks ]

Kaiser, T. & Menkhoff, L. (2016). Does financial education impact financial behavior, and if so, when? DIW Berlin Discussion Paper 1562 Retrieved from https://ssrn.com/abs- tract=2753510 or http://dx.doi.org/10.2139/ssrn.2753510Links ]

Kempson, E., Collard, S., & Moore, N. (2006). Measuring financial capability: An exploratory study for the Financial Services Authority (Consumer Research 39). Retrieved from http://www.pfrc.bris.ac.uk/publications/completed_research/Reports/Fincap_June05.pdfLinks ]

Kotlikoff, L. J. and Bernheim, B. D. (2001). Household financial planning and financial literacy. In L. J. Kotlikoff (Ed.), Essays on saving, bequests, altruism, and life-cycle planning (pp. 427-478). Cambridge, MA: MIT Press. [ Links ]

Jacobs, F. H. (1988). The five-tiered approach to evaluation: Context and implementation. Evaluating Family Programs, 37-68. [ Links ]

Johnson, E., & Sherraden, M. S. (2007). From financial literacy to financial capability among youth. Journal of Sociology and & Social Welfare, 34, 119. [ Links ]

Karlan, D., & Valdivia, M. (2011). Teaching entrepreneurship: Impact of business training on microfinance clients and institutions. Review of Economics and Statistics, 93(2), 510-527. [ Links ]

Lusardi, A., & Mitchell, O. S. (2005). Financial literacy and planning: Implications for retirement wellbeing. Michigan Retirement Research Center, WP 108. [ Links ]

Lusardi, A., & Mitchell, O. (2009). How ordinary consumers make complex economic decisions: Financial literacy and retirement readiness. NBER Working paper 15350. Retrieved from: http://www.nber.org/papers/w15350Links ]

Lusardi, A., & Mitchell, O. S. (2011). Financial literacy around the world: An overview. Journal of Pension Economics and Finance, 10(04), 497-508. [ Links ]

Lusardi, A., & Mitchell, O. S. (2014). The Economic importance of financial literacy: Theory and evidence. Journal of Economic Literature, 52(1), 5-44. [ Links ]

Lusardi, a., Mitchell, O., & Curto, V. (2010). Financial literacy among the young. Journal of Consumer Affairs, 44(2), 358-380. [ Links ]

Lyons, A. C., & Neelakantan, U. (2008). Potential and pitfalls of applying theory to the practice of financial education. Journal of Consumer Affairs, 42(1), 106-112. [ Links ]

MandeLL, L. (2008). The financial literacy of young American adults: Results of the 2008 National Jump$tart Coalition Survey of high school seniors and college students. Retrieved from www.jumpstart.org/assets/fiLes/2008SurveyBook.pdfLinks ]

Meier, S., & Sprenger, C. (2007). Selection into financial literacy programs: Evidence from afield study. Federal Reserve Bank of Boston Discussion paper 07-5. [ Links ]

Meier, S., & Sprenger, C. D. (2013). Discounting financial Literacy: Time preferences and participation in financial education programs. Journal of Economic Behavior and Organization, 95, 159-174. [ Links ]

MiLLer, M., Reichelstein, J., Salas, Ch., & Zia, B. (2014). Can you help someone become financially capable. A meta-analysis of the literature. PoLicy Research Working Paper No. 6745. Retrieved from: http://documents.worLdbank.org/curated/en/Links ]

Núñez, J., SaLazar, N., Castañeda, C., Rivas, G., Cuesta, L., & Castillo, J. (2012). Primer seguimiento de la evaluación de impacto del piloto del programa de promoción de cultura del ahorro. Mimeographed document. FedesarroLLo. Retrieved from: http://www.repository.fedesarroLLo.org.co/handLe/11445/185Links ]

OECD/INFE. (2012). High-LeveL PrincipLes on NationaL Strategies For FinanciaL Education. [ Links ]

OECD/INFE. (2016). OECD/INFE PoLicy Handbook - NationaL Strategies For FinanciaL Education. AntaLya, Turkey. [ Links ]

OECD. (2005). Improving Financial Literacy: Analysis of Issues and Policies. Paris: OECD PubLishing. [ Links ]

OECD. (2014). OECD/INFEprogress report. Tech. rep., Paris. Retrieved from: http://www.oecd.org/daf/fin/financial-education/OECD-INFE-Fin-Ed-G20-2014-Progress-Report.pdfLinks ]

PISA, O. (2012). Results in Focus: What 15-year-oLds know and what they can do with what they know. Retrieved from http://www.oecd.org/pisa/keyfindings/pisa-2012-results-overview.pdf. [ Links ]

Redy, R., Bruhn, M., & Tan, C. (2013). Capacidades Financieras en Colombia: Resultados de la encuesta nacional sobre comportamientos, actitudes y conocimientos financieros. Tech. rep., Banco MundiaL. [ Links ]

Robinson, J. a., & Torvik, R. (2005). White eLephants. Journal of Public Economics, 89(2-3), 197-210. [ Links ]

Rodríguez, C., Sanchez, F., & Zamora, S. (2014). On the radio: Effectiveness of the Viva Seguro financiaL education program. Serie Documentos Cede (2014-19). Obtenido de http://economia.uniandes.edu.coLinks ]

Van Rooij, M. C., Lusardi, A., & Alessie, R. J. (2012). FinanciaL Literacy, retirement pLanning and househoLd weaLth. The Economic Journal, 122(2006), 449-478. [ Links ]

Van Rooij, M., Lusardi, A., & Alessie, R. (2011). FinanciaL Literacy and stock market participation. Journal of Financial Economics, 101(2), 449-472. Retrieved from: http://dx.doi.org/10.1016/j.jfineco.2011.03.006Links ]

Xu, L., & Zia, B. (2012). Financial literacy around the world: An overview of the evidence with practical suggestions for the way forward. (Policy Research Working Paper 6107). Retrieved from: http://documents.worldbank.org/curated/en/264001468340889422/Financia-literacy-around-the-world-an-overview-of-the-evidence-with-practical-su-ggestions-for-the-way-forwardLinks ]

Yoong, J., Mihaly, K., Bauhoff, S., Rabinovich, L., & Hung, A. (2013). A Toolkitforthe Evaluation of Financial Capability Programs in Low-, and Middle-Income Countries. Washington DC: World Bank. Retrieved from https://openknowledge.worldbank.com/handle/10986/16294Links ]

*doi: 10.11144/Javeriana.cao30-54.caef. Este artículo es producto de la investigación "Metodologías para medir alfabetización financiera", financiado por la Universidad EAFIT, desde enero de 2015 hasta abril de 2016. El artículo se recibió 26-10-2015 y se aprobó el 10-03-2017. Sugerencia de citación: Álvarez-Franco, P., Muñoz-Murillo, M., & Restrepo-Tobón, D. (2017). Challenges in assessing the effectiveness of financial education programs: The Colombian case. Cuadernos de Administración, 30(54), 155-180. http://dx.doi.org/10.11144/Javeriana.cao30-54.caef.

1 Fox, Bartholomae, and Lee (2005) document that several financial education initiatives developed in the United States since the 1990's failed to include an evaluation component in their design. Thus, most of these programs offered few insights regarding their effectiveness on improving financial literacy or financial outcomes.

2 The countries designing a national strategy are: Argentina, Canada, Chile, China, Colombia, Costa Rica, Croatia, El Salvador, France, Guatemala, Kenya, Lebanon, Malawi, Mexico, Pakistan, Peru, Poland, Romania, Saudi Arabia, Serbia, Sweden, Tanzania, Thailand, Uganda, Uruguay, and Zambia. The countries implementing one: Armenia, Brazil, Czech Republic, Denmark, Estonia, Ghana, India, Indonesia, Ireland, Israel, Korea, Latvia, Malaysia, Morocco Nigeria, Portugal, Russian Federation, Slovenia, and Turkey. The countries revising it: Australia, Japan, Netherlands, New Zealand, Singapore, Spain, South Africa, United Kingdom, and United States.

3 There could be other causes of financial literacy. Lusardi and Mitchell (2009) find that studying economics in high school is associated with higher levels of financial literacy. Christiansen, Joensen, and Rangvid (2008) find that studying economics in college is causally related to holding stocks.

4 Cohen (1977) suggested, as a rule of thumb, that effect sizes smaller than 0.20 should be considered "small", effect sizes around 0.50 "medium" and greater than 0.80 "large".

5 The endogeneity in financial literacy and financial outcome studies could arise from an error of measurement in the independent variable, a simultaneity between the independent and the dependent variable, or an omitted variable correlated with the independent variable (Hill, Griffiths, & Lim, 2008).

6 For a step-by-step description of power calculation see Gertler et al. (2011).

7 Comisión Intersectorial de Educación Económica y Financiera (2016). The authors thank Nidia Garcia Bo- hórquez, Chief of Economics and Financial Education Section - Colombia Central Bank, for her willingness to help us in this research.

8 Informal funds created with resources of relatives, friends, and neighbors over the years and distributed at the end of the period.

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License