Mathematica’s #EvidenceInsight Video Series: Cutting-Edge Methods in Policy Research

In the face of proliferating data sources and research technologies, budget cutbacks, and the growing prominence of performance-based programs and evidence-based policy-making, there has been an increase in demand for faster, cheaper program evaluation. To address this need, the Center for Improving Research Evidence (CIRE) at Mathematica Policy Research has produced the Evidence Insight video series featuring four innovative research methods by different Mathematica researchers.  The goal of the series is to make policy makers, researchers and practitioners aware of the strengths, limitations and contexts of these new evaluation techniques and enable continuous improvement of program delivery and effectiveness.

Summaries of the transcribed video of each of these techniques follow:

Rapid-Cycle Evaluation: Determining What Works in Less Time

Small changes to program operations and service delivery can improve the efficiency and effectiveness of the program. This video delivered by Alex Resch of Mathematica, describes how rapid-cycle evaluation can quickly determine the impact of these changes. Such evaluations rely predominantly on existing program data and rigorous research methods to measure impacts of changes on program delivery, to identify causal links between the changes and the outcomes, and to focus on results that can be observed in a short time. Because rapid-cycle evaluations proceed relatively quickly, the results give decision makers and practitioners the timely and reliable evidence needed for continual program improvement.

CIRE #EvidenceInsight Video Series – Rapid Cycle Evaluation from Mathematica Policy Research on Vimeo.

Bayesian Methods: A Faster, Probabilistic Approach to Research Design

Bayesian methods allow researchers to rigorously use external data to supplement traditional data collection sources and techniques to gain insights about programs and policies that would not have been possible with the traditional evaluation techniques alone. In this video, Lauren Vollmer of Mathematica explains how these methods produce precise probabilities, enabling policymakers to target programs and policies to meet the needs of specific audiences. Moreover, the increased precision enables detection of meaningful affects even when sample sizes are small, thus reducing the cost of data collection. And the ability to draw conclusions in probabilistic terms along a spectrum offers researchers and policymakers greater opportunity to apply subject-matter expertise in determining whether effects are meaningful in context. While there are costs involved, such as sometimes unreliable external data and computational complexity, these obstacles have been met by the development of comprehensive guidelines and sensitivity checks as well as advances in computation that enable faster and less computationally draining Bayesian analysis.

CIRE #EvidenceInsight Video Series – Bayesian Methods from Mathematica Policy Research on Vimeo.

Adaptive Randomization: A Fresh Perspective on Traditional Research Design

Randomized control trials (RCTs) are considered to be the gold standard for providing valid and rigorous evidence of program evidence by controlling for non-program factors such as the characteristics of program participants that possibly contributed to observed outcomes. However, they can be time-consuming and expensive to implement. In this video, Mariel Finucane and Ignacio Martinez discuss adaptive randomization, an innovative extension of traditional RCTs. Faster and cheaper than RCTs, adaptive randomization enables researchers to adjust their methods in response to accumulating evidence without losing the methodological principles of randomization. This new research method, when combined with access to rich datasets such as electronic health records, contributes, in efficient and rigorous fashion, to our understanding of how specific program elements can be refined to better meet the needs of particular subgroups.

CIRE #EvidenceInsight Video Series – Adaptive Randomization from Mathematica Policy Research on Vimeo.

Predictive Analytics: Transforming Decision Making in Three Steps

Predictive analytics is an emerging research field in which data and evidence are used to predict outcomes and inform decision making. In this video, Phil Killewald of Mathematica breaks the predictive process down into three steps:

  • Define the problem: gain thorough knowledge of the extent and quality relevant data bases and of agency activities; and pose evaluation questions to address shortcomings in program effectiveness;
  • Design the predictive model: gather historical data on past activities where a shortcoming is to be addressed; design, build and evaluate a number of predictive models using these data; and test the model using separate data in order to identify the best performing models; and
  • Make the predictions: use the best performing model to predict outcomes with the real-world data where the true outcomes are not yet known, make the model available as early as possible, and periodically assess its accuracy and update it as necessary or as new data become available.

When applied to policy-relevant research questions, predictive analytics can be a valuable tool for organizations seeking to improve their services and their decision making processes by identifying what works for whom under what conditions on the basis of evidence-based outcome predictions.

CIRE #EvidenceInsight Video Series – Predictive Analytics from Mathematica Policy Research on Vimeo.

This information should increase awareness of and amenability of these new techniques among not only researchers, but also practitioners seeking to improve the effectiveness of the outcomes they deliver.

29 Comments

  1. Hello! I realize this is kind of off-topic however I had to
    ask. Does operating a well-established blog such as yours require a large amount of work?
    I’m brand new to writing a blog but I do write in my journal on a daily basis.
    I’d like to start a blog so I will be able to share
    my own experience and thoughts online. Please let me know if you have any
    kind of ideas or tips for new aspiring blog owners. Appreciate it!

  2. Hey there are using WordPress for your site platform?

    I’m new to the blog world but I’m trying to get started and set up
    my own. Do you require any html coding expertise to make your own blog?
    Any help would be really appreciated!

  3. I just like the valuable information you supply
    in your articles. I’ll bookmark your weblog and take a look at again right here frequently.
    I’m relatively certain I’ll learn many new stuff right here!
    Good luck for the next!

  4. I was wondering if you ever considered changing the structure
    of your website? Its very well written; I love what youve got to say.
    But maybe you could a little more in the way of content so people could connect with it better.
    Youve got an awful lot of text for only having 1 or 2 pictures.

    Maybe you could space it out better?

  5. Do you have a spam issue on this website;
    I also am a blogger, and I was wondering your situation; many of
    us have developed some nice methods and we are looking to exchange
    techniques with other folks, why not shoot me an email if interested.

  6. naturally like your website but you have to check the
    spelling on quite a few of your posts. A number
    of them are rife with spelling problems and I in finding it very troublesome
    to tell the reality on the other hand I will
    surely come back again.

  7. Hi there, I discovered your web site by way of Google while searching for a comparable
    topic, your web site got here up, it seems good.
    I’ve bookmarked it in my google bookmarks.

    Hi there, just become aware of your weblog via Google, and located that it’s really informative.
    I’m going to be careful for brussels. I’ll be grateful if you proceed this in future.
    Many other people might be benefited from your writing. Cheers!

  8. You can definitely see your skills within the article you write.
    The arena hopes for even more passionate writers like you
    who are not afraid to say how they believe. Always follow your heart.

  9. Normally I do not read post on blogs, but I would like to say that
    this write-up very compelled me to try and do
    so! Your writing taste has been amazed me. Thanks, very great
    article.

  10. Great goods from you, man. I’ve bear in mind your stuff prior to and you are simply too fantastic.
    I actually like what you’ve got right here, really like
    what you’re stating and the way in which through which you assert it.
    You’re making it enjoyable and you still care for to
    stay it smart. I cant wait to read much more from you.
    This is really a great site.

Join the Conversation

Your email address will not be published. Required fields are marked *