“When we enter our clients’ homes, we’re not just nurses, we’re also confidants and counselors,” said nurses from Nurse-Family Partnership’s DuPage County, Illinois local agency. For decades, Nurse-Family Partnership (NFP), a national non-profit organization that sends specially-trained nurses into the homes of young, low-income, first-time mothers until their babies are 2 years old, has demonstrated its impact on maternal health and education, early childhood development, and economic self-sufficiency through randomized controlled trials.
Now, 18 years after first rolling out the NFP program to local communities nationwide, NFP serves more than 29,500 families in over 550 counties. These numbers are set to increase thanks to increased federal funding over the past few years from the Maternal, Infant and Early Childhood Home Visiting Program.
Yet, even as NFP looks to expand in new cities, some existing local agencies are facing greater demand than they can meet. Hence, this summer, our team — fellows Sarah Abraham, Jeff Lockhart, Sarah Tan, Rafael Turner, and mentor Young-Jin Kim — is helping NFP take a hard look at its operations and how its program is being implemented in local communities nationwide.
A Typical NFP Mom
Today, NFP serves a diverse population of mothers across 43 states. However, what all of these mothers have in common is that they are young, first-time, low-income mothers. The majority of them are unmarried and many of them are 19 years old or younger. Hence, this is a highly vulnerable population.
To Define the “Undefinable” and Measure the “Unmeasurable”
When NFP first came to us at the beginning of this summer, the task was simple, or so we thought — identify mothers who will “benefit the most” from the NFP program and the most impactful timing of nurse visits, in hopes that these would help NFP better understand their target population and personalize their services based on each mother’s needs.
We quickly realized the folly in our thinking. Unlike for-profit organizations, where the definition of success is frequently clear-cut and usually measured in monetary terms — profit, customer base, share prices — non-profit organizations such as NFP have to first figure out what they are trying to achieve and pick metrics that map to these outcomes. Then, they must measure the “unmeasurable,” an endeavor that is complicated by the challenge of determining exactly how much of these outcomes can be attributed to them and how much would have happened anyway independently of their efforts.
This last step, frequently called impact evaluation, was exactly what last year’s team helped NFP with. By combining NFP’s data with national demographic data, last year’s team generated a counterfactual population to assess how nurse home visits affect measures of early childhood development such as immunization and breastfeeding rates. The results of their analysis will be used come spring to support NFP’s advocacy of an extension of government funding through the renewal of the federal home visiting program.
This year, the need to demonstrate measurable improvements in benchmark areas in order to continue justifying federal funding has switched the focus back on the first two steps — defining success, and shaping metrics to measure success. Hence, we set about exploring ways of defining the “undefinable”.
When and Why Do Moms Leave the Program?
We quickly found our journey fraught with uncertainty, and decided to research why women leave NFP before their children’s second birthdays. A natural first step might be to consider retention as a metric of success. However, if a NFP mother stays to the end of the 2.5-year program, is that truly success? How about a mother who only stayed for 1 year, yet managed to achieve all of NFP’s program goals? This mother arguably saved NFP money and freed up space for another mother to enter the program.
Fortunately, in the exit interview when an NFP mom leaves, she is asked whether she received what she needed from the program. If so, one could say that she left the program for a “good reason.” A mother could also leave early for a “neutral” reason, such as a local agency closing, or an unfortunate miscarriage. We hope that augmenting when the mother leaves the program with why she leaves gives us a more nuanced look at retention
Regardless of when or why the mother leaves the program, we could also consider whether or not the mother achieved NFP’s program goals of improved maternal health and education, early childhood development, and economic self-sufficiency — all of which are measured throughout the program. The following graph illustrates the outcomes attained by mothers who stay in the program up to that point where the metric is measured.
By considering multiple metrics of success, we hope to paint a more complete picture of an NFP mother’s experience in the program.
A Multidimensional Look at Success
Exploring these lines of thought motivated us to investigate NFP’s operations in a variety of different ways. We are eager to learn when and why mothers leave the program, how satisfied mothers are with the program, and what helps mothers achieve program goals. Then, once we learn all of these, we want to help NFP design experiments to test the findings of our models. If our models speak the truth, we hope to help NFP develop targeted interventions that reduce the attrition rate for mothers who still stand to gain from the program and target mothers who need the extra push to achieve program goals.
“NFP accrues considerable data on the nature of our services and the outcomes that our clients achieve,” said Dr. Bill Thorland, NFP’s national program evaluator. “Working with DSSG provides an exciting opportunity to bring fresh eyes to problems of interest to our program operations.”
We are excited to help NFP as it starts utilizing data collected from local communities, beyond their randomized controlled trials, to optimize its operations.