About Evaluation and Research


Evaluation is a process used to improve the design and function of educational experiences in informal learning environments and to measure progress toward goals. Research studies are designed to generate findings that will inform the broader field of informal science education.

Evaluation

Research

Evaluation

hand writing evaluation sticky notes

The NISE Network Evaluation group is a multi-organizational team dedicated to informing and improving the work of the Network by conducting evaluation studies that closely examine the public, professional, and field-wide impacts of NISE Net activities. For the first five years of the Network, group members worked closely with nearly all NISE Net groups and teams, including Administration, Exhibits, Forums, NanoDays, Network Community, Network Media, Researcher-Informal Science Education (RISE), and the website group, amongst others. In each area, evaluators worked with the Network teams, conducting studies that encouraged professional inquiry and provided constructive feedback. For the second five years of the Network, the Evaluation group has focused on conducting summative evaluations to understand the impact of the NISE Network. This has included studies on the products that have the greatest impact on the public, such as the Nano Mini-exhibit and NanoDays, as well as a three-year study looking at the impact of the Network as a whole on the informal science education professionals who are involved. In addition, the Evaluation group has continued helping other Network teams to gather data to inform their own work through a process called Team-Based Inquiry. Over the course of the NISE Net, over 250 front-end, formative, and summative evaluation studies have been conducted about NISE Net products and practices. 

To document some of the challenges we faced and lessons we learned about evaluating one of the largest informal education networks ever formed, we created a reflection document that can be found here:

We also documented how our team used an Evaluation Committee of Visitors (COV), or external review panel, to support and oversee the internal Evaluation workgroup. Our reflections on evaluation advisory committees can be found in this one-pager: 

Network-Wide Evaluation

Summative Evaluation iconSummative Evaluation Studies

Summative evaluation typically occurs at the end of a project to assess the impact of the completed project. This process usually includes observing visitors while they experience a program or exhibit and interviewing or surveying them before and/or after this experience.The NISE Network Evaluation Team has conducted several summative studies investigating issues related to public and professional impacts across the Network.

Below is a list highlighting a few of the most recent NISE Net summative evaluation reports. The full list of summative NISE Net evaluation work is available here:

  • Explore Science: Earth & Space Activity Toolkits Summative Study (2019): This summative evaluation study of the toolkits was designed to address and answer questions regarding the number and range of audiences reached by the project, visitor engagement, relevance, science identity, and their understanding around content areas.

  • Building with Biology Participant Impact Evaluation Report (2018): This report focuses on outcomes for public audiences who engaged in public events with hands-on activities and forum programs about synthetic biology as part of the Building with Biology project.

  • Multi-Site Public Engagement with Science - Synthetic Biology Final Evaluation Report (2018): This report focuses on outcomes for the scientists and informal science educators involved as event hosts and/or facilitators in the Building with Biology project.
  • NISE Net Museum & Community Partnerships Project Evaluation Report (2017): This study focused on NISE Net’s efforts to support collaborations between museums and local community organizations, investigating impacts on professional participants as well as the extent to which the project reached underserved audiences with relevant, educational, and engaging materials.

  • NISE Net Years 6‐10 Evaluation Summary Report (2017): The purpose of this report is to summarize the findings from the Years 6-10 (2010-2015) Nanoscale Informal Science Education Network (NISE Net) summative evaluation studies and to discuss factors that contributed to the achievement of NISE Net’s goals.

  • NISE Network Professional Impacts Summative Evaluation (2016): This investigation is based on the NISE Network goals for professionals and explores how involvement with NISE Net impacts an individual professional’s sense of community, learning about nano, and use of nano educational products and practices.
  • NISE Network Public Impacts of Nano-Rich Organizations Summative Evaluation (2016): This study was designed to explore how the NISE Network’s range of resources were used by the diverse range of Network partners, and how individual institutions felt that they were doing at providing their publics with positive experiences and learning outcomes.
  • NISE Network Public Impacts Summative Evaluation (2012-2015): Several studies provide an overview of the Network's overall impacts on the public.
  • Public Reach Estimations for the NISE Network (2015): The purpose of this document is to consolidate and archive all of the major public reach estimates. Over the life of the project, the NISE Network is estimated to have reached over 30 million people through NanoDays and the Nano exhibition.
  • Summative Evaluation Study of NanoDays 2014 Events: In the spring of 2014, the Nanoscale Informal Science Education Network (NISE Net) Public Impacts evaluation team conducted a summative study of NanoDays, a nationwide festival of educational programs about nanoscale science, engineering, and technology. 
  • The Public Impacts Mini-Exhibition Study Year 8 Summative evaluation (2013) focused on the reach of the Nano mini-exhibition and whether or not the exhibit was successful in providing visitors with an engaging and educational experience. This study also analyzed the mini-exhibition’s success in different contexts and for different types of audiences, including Hispanic visitors and visitors with disabilities, and whether or not it was a catalyst for new public programming around nano.
  • A Year 6 Network Communication Study was conducted to learn about how the Network's primary communication components (NanoDays, face-to-face meetings, the regional hub structure, and the nisenet.org website) are being used by actively involved partners.
  • The Exhibits and Programs Year 5 Summative Evaluation studied the impact of a set of NISE Net exhibits and programs on museum visitors.
  • The Year 5 Delivery and Public Reach Study examined nano educational activities that NISE Net partners are doing and estimated the public reach of the Network in 2010.
  • Review of NISE Network Evaluation Findings: Years 1-5  is a review of the findings from over 240 evaluation reports from the first five years of the Network. The review is divided into six chapters:
    • Connecting ISE Professionals with Nano Informal Science Education
    • Connecting the University-Affiliated Individuals with Nano Informal Science Education
    • Engaging the Public in Learning about Nano through NISE Network Educational Products
    • Engaging the Public with Societal and Ethical Implications Content through NISE Network Products
    • Making the Unfamiliar Interesting and Relevant for Museum Visitors
    • Reaching Public Audiences

 Other Evaluation Studies

The NISE Net Evaluation Team has also conducted front-end and formative studies in order to guide the development of most NISE Net products and practices.

Front-End Evaluation iconFront-End Evaluation

During the early stages of a project, front-end evaluation is used to learn about visitors’ familiarity with a topic, their interests and feelings, and what understandings they bring to an experience. Examples of front-end studies for NISE Net have included reviews of existing projects about nanoscience and a study investigating public awareness, interest, knowledge, and attitudes related to nanotechnology. Links to these studies are below:

 

 Formative Evaluation icon​Formative Evaluation

During the design and development stage of a project, formative evaluation is used to help achieve desired goals; formative evaluation involves iterative testing and modification of prototype exhibits, materials, and programs.

In the first five years of the Network, all formative evaluations were conducted by members of the Evaluation Team and included several rounds of testing of exhibit prototypes, program work such as forums and stage presentations, along with interactive media products. In the second five years of the Network, formative evaluations were conducted by NISE Net practitioners through a Team-Based Inquiry process.

Below are examples of formative studies that were conducted by the NISE Net Evaluation group in the first five years of the network.

  • Who Decides Forum is a formative evaluation that looked at ways to improve visitor enjoyment, engagement and perceived learning at a discussion forum that was held at five institutions.
  • Intro to Nanotechnology is a formative evaluation that explored the effectiveness of an initial set of NISE Net prototype exhibits in 2006.
  • Exploring Measurement-Stretchability is a formative evaluation that looked at children and adults’ engagement and comprehension of a nano activity focused on the nanoscale.
  • Treating Tumors with Gold is a formative evaluation that gathered visitors’ feedback to a stage presentation on research using nanoshells and infrared light in cancer therapy.

More NISE Net evaluation reports are available within the following section of the website. Where possible, reports are also included as resources for each of the related products.

Team-Based Inquiry

Team-Based Inquiry cycle

Team-Based Inquiry (TBI) is a practical approach to empowering education professionals to get the data they need, when they need it, to improve their products and practices and, ultimately, more effectively engage public and professional audiences. The TBI process involves an ongoing cycle of inquiry: question, investigate, reflect, and improve.

  • The Team-Based Inquiry Guide explains each step of the TBI process and features ways TBI is used in the NISE Network to improve educational experiences and professional practice. 
  • The Team-Based Inquiry Training Videos were designed to support museum practitioners in learning about and using TBI in their own institutions by describing this process and providing examples of how a team might work through the different steps. 
  • NISE Network Program Evaluation Tools includes guidelines and templates to facilitate program evaluation.

Who We AreEvaluators in action

Different projects have included staff members from the following organizations.
If you have any questions or want further information about the evaluation and research, please contact Liz Kollmann (ekollmann@mos.org) or Marta Beyer (mbeyer@mos.org).

Space and Earth Informal Science Education (SEISE) project

  • Museum of Science, Boston
  • Science Museum of Minnesota
  • External committee of Visitors: Gina Svarovsky, University of Notre Dame

Nanoscale Informal Science Education Network project

Over the course of the NSF-funded Nanoscale Informal Science Education Network project, the evaluation team has included staff members from the following organizations:

Years 1-5:

  • Museum of Science, Boston
  • Science Museum of Minnesota
  • Oregon Museum of Science and Industry
  • Exploratorium
  • Inverness Research Associates
  • Multimedia Research

Years 6-12:

  • Museum of Science, Boston
  • Science Museum of Minnesota
  • Oregon Museum of Science and Industry

Committee of Visitors
In addition, the Nanoscale Informal Science Education Network summative evaluation work for years 5-12 has been overseen by an external Committee of Visitors. 

  • Frances Lawrenz
  • Bruce Lewenstein
  • Saul Rockman
  • Carol Weiss was also a member of the Committee of Visitors for several years.

NISE Network's online surveys are powered by SurveyGizmo.

Research

The Nanoscale Informal Science Education Network is conducting several multi-year research projects looking at different aspects of the Network that have the potential to inform the field of informal science education. Whereas the evaluation studies conducted by the NISE Network measure progress toward NISE Net goals, the research studies are designed to generate findings that will inform and advance the broader field of informal science education. Below is a description of these projects. For more details, contact members of the research team.

Research Report icon

Research Reports on the website

Research reports summarize the results of research studies conducted on the impacts of informal science education efforts on both public audiences and museum professionals.

Below are examples of formative studies that were conducted by the NISE Net Evaluation group in the first five years of the network.

 

1. Museum-Scientist Partnerships:

This project examines the nature and impacts of the development of partnerships between informal science institutions and practicing scientists.  We are interested in learning how museum-scientist partnerships create useful and meaningful representations of complex and cutting-edge scientific and socioscientific concepts; how these representations underpin the development of informal education materials, experiences and approaches that are used by museum exhibit designers and program staff; and how new products and practices resulting from museum-scientist partnerships are used in supporting and expanding the NISE Network. A case study follows the formation of the Nano & Society partnership between museum educators and scientists through its inception to the four regional workshops given on the topic. Another study, the network analysis of survey data, provides insightful information about connections between organizations within the NISE Network.

2. Organizational Change:

This study examines whether and how organizations affiliated with NISE Net experience widespread and long-lasting change in their practices. Previous studies from formal education and other fields have shown that networks can serve as a powerful impetus for change as they enable the introduction of new practices and ideas into an organization through frequent interactions with other similar organizations. This study looks at partners and explores the conditions that facilitate or impede the adoption of NISE Net-related knowledge and practices within these organizations.

3. Public Learning and Decision Making:


This project focuses on exploring the different ways NISE Net can engage visitors in learning and decision making about nano. Currently, this project is studying how visitors use, interact with, and talk about the different exhibit components within the Nano mini-exhibition to learn about the different areas of the NISE Network content map. In particular, the team will research how visitors’ understandings of relevance, or connection to their lives, develops within the exhibition.

4. Tracking NISE Net’s Real World Impact:

This project examines media coverage and online discourses about nanotechnology, with a particular focus on NISE Network-related work, through tracking media content, blog posts, and discussions on various social media platforms (e.g., Facebook and Twitter). We use combinations of commercial software packages, including Lexis Nexis, Vantage Point, and Crimson Hexagon, to track large amounts of communication offline and online in a real time fashion.

First, we track the volume of coverage in traditional media channels and discussions on various online platforms. Second, we monitor genuine and spontaneous responses on nanotechnology-related topics among concerned publics in online environments. This project tracks sentiment, expressed with different levels of certitude, and thematic content such as risk-benefit assessment, of nanotechnology-related topics, especially NISE Network activities. Third, we examine the relationship between NISE Network events across different regional sites and the traffic of relevant discussions on both traditional and online media channels. For instance, this project will track online traffic surrounding particular sites before and after events such as NanoDays or other NISE Network coordinated activities, and explore the relationship between the tracked traffic and the corresponding geographical origins of online discussion (such as Tweets).