Assets for Independence Evaluation

Overview

The U.S. Department of Health and Human Services, Administration for Children and Families (ACF) is conducting an experimental evaluation of the Assets for Independence (AFI) Program. This evaluation-the first experimental evaluation of Individual Development Account (IDA) projects operating under the Assets for Independence Act-will contribute importantly to understanding the effects of IDA projects and IDA project features on participants. IDA programs provide matching funds to participants when the savings are withdrawn to spend on qualified asset purchases, most commonly homeownership, business-related expenses, or education. This study will build on the prior quasi-experimental AFI evaluation, as well as studies of other non-AFI funded Individual Development Account (IDA) projects. While some evaluations suggest that IDAs help low-income families save, rigorous experimental research is limited. No experimental evaluation of AFI-funded programs has been conducted to date. Of the quasi- or non-experimental studies, few have focused on AFI-funded IDAs. The evaluation is being conducted in two sites, with the random assignment of 299 AFI-eligible cases in one site and 509 cases in the other, for a total of 808 cases. Each site randomly assigned sample members to one of two groups: a control group and a treatment group receiving conventional AFI services. The primary research questions the study seeks to answer are: 1. What are the impacts of AFI program participation on the outcomes it is designed to address (e.g., savings and asset purchases)? 2. What are the impacts of AFI program participation on other outcome domains that the program might influence (e.g., material hardship, alternative financial product use, economic well-being)? 3. How do the impacts of AFI program participation vary over time and by participant characteristics? The study uses a two-armed (treatment vs. control) experimental design. This design allows the first two questions above to be addressed using a rigorous randomized approach. In addition to the impact study, the evaluation includes an implementation study to describe and document how the AFI program is designed, implemented and operated in the participating grantee sites during the time of the study. This information will be used to provide context for interpreting the findings of the impact study.

The Assets for Independence (AFI) evaluation is being conducted in two sites. Both sites are located in major metropolitan areas of the southwestern U.S., serving client populations that are 60-75 percent Hispanic. Both have some similar program features: maximum savings periods of 24 months and matchable savings amounts of up to $1,000. Important for estimating design feature differences, there are also key differences in program parameters: required hours of financial education and IDA match rates. The sites enrolled their samples within a 19-month period from January 2013 through July 2014. Sample members in each site were randomly assigned to two equal-sized groups: a treatment group (those able to enter the site's IDA program) and a control group (those not able to enter). IMPACT STUDY The impact study is the primary quantitative component of the research. The data collection and analysis will provide estimates of the effect of IDA participation on short- and long-term client outcomes, as measured through 12-month, 36-month, and 60-month follow-up periods. Participant outcomes will be measured at the time of random assignment and 12, 36, and 60 months after random assignment. Measures that pertain to short-term program effects include amount of savings and material hardship avoidance. These are the effects most likely to emerge within the first year. Additional measures include employment status, earnings, means-tested public benefit receipt, and net worth. In the long-term, effects on homeownership, business ownership, and secondary education attainment are likely to emerge. An unexplored area of short-term effects of IDAs is the use of unmatched savings to meet emergency needs. In the early stages of AFI participation, households may establish a savings habit but may need to draw on their savings for unexpected expenditures or to offset an unanticipated drop in income. The hypothesized effect of such unmatched withdrawals of IDA deposits is an avoidance of material hardship in food and health. This effect will be estimated using a small set of household-level survey questions, including some items drawn from the adult well-being topical module of the Survey of Income and Program Participation (SIPP). RTI International has developed a web-based sample enrollment tool for IDA project staff at each site to use in administering informed consent, a baseline questionnaire, and random assignment for each study-eligible applicant. The study's 30-minute baseline questionnaire gathered information in the following survey modules: demographics, use of financial services, current residence, employment, household income and expenses, financial experience and assets, debts, material hardship, social engagement and outlook, and locating information (for purposes of re-contacting the respondent for follow-up surveys). Site-provided administrative data will provide additional baseline covariates and (for treatment cases) descriptive information on IDA openings, deposits, withdrawals (matched and unmatched), and balances. Note that person-level measures are specific to the "primary household member": the first individual from his/her household to apply for the IDA program during the study enrollment period. This person is the respondent for the baseline questionnaire and follow-up surveys. If other (secondary) members of the same household subsequently apply to enter the site's IDA program, they are not separately subject to random assignment (and do not add to the sample count) but are regarded as having been assigned to the same sample group ( treatment or control) as the primary member. Household-level outcomes are measured for the household to which the primary member belongs. The follow-up surveys obtain outcome and other contextual measures. The surveys take 30 minutes and are conducted via telephone, web, or an in-person interviewer. The respondent is the primary sample member, the same individual who applied to enter the AFI project and who completed the informed consent form and baseline questionnaire. The surveys obtain data on outcome measures, within the same topical headings as shown earlier for the baseline questionnaire. The estimation of program impacts will use conventional multivariate regression techniques as appropriate to the statistical properties of the dependent variable (continuous, dichotomous, or multinomial). For each outcome measure, the key parameter to be estimated is the coefficient on the randomly assigned (treatment versus control) status of the case. This coefficient indicates the regression-adjusted treatment-control difference in the measured outcome. All estimating equations will include covariates measured through the baseline questionnaire. The program effect estimated is the intent-to-treat (ITT) effect, comparing the outcomes between control group cases and all treatment group cases, regardless of whether the treatment case opens an IDA account. Stated otherwise, this procedure yields an estimate of the effect of the offer of IDA participation, as not all treatment group members will open IDAs. We will also estimate the treatment-on-the-treated (TOT) effect (i.e., the impact of the program on those who complete it). In addressing the first research question--the impact of AFI participation on the outcomes it is designed to address (e.g., savings and asset purchases)-we will estimate the program effect separately by site as well as from the pooled sample combining data from both sites. The pooled sample data will provide the study's most precise single estimate of the program effect, fully utilizing all available sample observations. The second research question-regarding the effect on other outcome domains that the program might influence (e.g., material hardship, alternative financial product use, economic well-being)-will also be addressed by estimating effects on the pooled sample and separately by site. Under research question 3 we will test for differential program impacts by participant subgroups. The subgroups of interest will be identified from the baseline data. We anticipate these subgroups will reflect such differences as age, initial income level, initial indebtedness, initial education level, or initial homeownership status. IMPLEMENTATION STUDY The implementation study will describe and document how the AFI project is designed, implemented, and operated in the participating grantee sites. This information will provide the context for understanding findings from the impact analysis. The main data collection approach involved site visits to the AFI grantees that included interviews with key administrators, staff, and stakeholders; observations of grantee services; and reviews of relevant documents and data. The site visits to the selected AFI grantees were two days each and occurred during 2014. While on site, we conducted semi-structured in-person interviews with individuals in differing roles to obtain a range of perspectives on the AFI project, including IDA grantee staff, financial education providers, and representatives of partner organizations. The interviews examine issues related to the design and implementation of the AFI project, challenges projects face and their strategies for overcoming them, and other factors that may not be captured in the quantitative analysis but that may affect project outcomes. Questions about design and implementation draw from information gathered during site selection, site training, initial random assignments, and monitoring calls, and cover whether, how, and why project changes have occurred. Through site selection and the on-site interviews with staff, we obtained information on organizational stability and capacity, level of IDA project complexity, number of staff and amount of staff involvement, and participant recruitment. Specific questions included: What outreach activities did you conduct to inform individuals of your IDA project? What forms of program support are provided to IDA accountholders? How often are AFI project staff in touch with participants after they open an account? What non-IDA program services were available to members of the comparison group through your organization? Do you feel that that there are any important effects (positive or negative) of your AFI project on participants beyond the measured outcomes we are tracking through the evaluation? In addition to asking respondents about the challenges they face, interviews with IDA project administrators and staff probe further on challenges such as helping participants attain realistic savings goals, navigating the requirements of diverse funding sources and requirements, raising nonfederal funds, and coping with limited funds for administrative costs. The interviews also explore external factors or special circumstances that may affect project outcomes. This information will be vital both for interpreting the impact analysis findings and for the generalizability of the findings to other sites. It will be fundamentally important to understand why a program has resulted in its estimated effects on participants and whether such effects could be replicated elsewhere. Finally, the interviews give respondents an opportunity to identify intangible outcomes, whether positive or negative, that they feel are significant but that may not be captured in the formal impact analysis, and help identify participant stories to share. During the site visits, we observed financial education classes and other project-participant interactions to capture what is actually happening during project service delivery. Observations were guided by predetermined topics that map to the specific study issues, and observers were as unobtrusive as possible. The observations will provide insights into how particular project aspects may influence participant outcomes. The implementation study also will include a document review to understand how the AFI project administrative structure and project design may influence project outcomes. Written documentation of the number and roles of staff and volunteers involved in the IDA project and in the organization itself, project rules, and overall project goals and objectives will set the context for identifying variables that could affect participant outcomes. Project communication materials provided to project participants, such as recruitment flyers, orientation materials, and program welcome letters, will be collected to understand project messaging. In addition, we are obtaining data that the projects are collecting, such as data on service take-up.