Evaluation is a process used to improve the design and function of educational experiences in informal learning environments and to measure progress toward goals. Research studies are designed to generate findings that will inform the broader field of informal science education.
- Network-Wide Evaluation
- Who We Are
- Museum-Scientist Partnerships
- Organizational Change
- Public Learning and Decision Making
- Tracking NISE Net’s Real World Impact
The NISE Network Evaluation group is a multi-organizational team dedicated to informing and improving the work of the Network by conducting evaluation studies that closely examine the public, professional, and field-wide impacts of NISE Net activities. For the first five years of the Network, group members worked closely with nearly all NISE Net groups and teams, including Administration, Exhibits, Forums, NanoDays, Network Community, Network Media, Researcher-Informal Science Education (RISE), and the website group, amongst others. In each area, evaluators worked with the Network teams, conducting studies that encouraged professional inquiry and provided constructive feedback. For the second five years of the Network, the Evaluation group has focused on conducting summative evaluations to understand the impact of the NISE Network. This has included studies on the products that have the greatest impact on the public, such as the Nano Mini-exhibit and NanoDays, as well as a three-year study looking at the impact of the Network as a whole on the informal science education professionals who are involved. In addition, the Evaluation group has continued helping other Network teams to gather data to inform their own work through a process called Team-Based Inquiry. Over the course of the NISE Net, over 250 front-end, formative, and summative evaluation studies have been conducted about NISE Net products and practices.
To document some of the challenges we faced and lessons we learned about evaluating one of the largest informal education networks ever formed, we created a reflection document that can be found here:
Summative evaluation typically occurs at the end of a project to assess the impact of the completed project. This process usually includes observing visitors while they experience a program or exhibit and interviewing or surveying them before and/or after this experience.The NISE Network Evaluation Team has conducted several summative studies investigating issues related to public and professional impacts across the Network.
Below is a list highlighting a few of the most recent NISE Net summative evaluation reports. The full list of summative NISE Net evaluation work is available here:
- NISE Network Professional Impacts Summative Evaluation: This investigation is based on the NISE Network goals for professionals and explores how involvement with NISE Net impacts an individual professional’s sense of community, learning about nano, and use of nano educational products and practices.
- Public Reach Estimations for the NISE Network: The purpose of this document is to consolidate and archive all of the major public reach estimates. Over the life of the project, the NISE Network is estimated to have reached over 30 million people through NanoDays and the Nano exhibition.
- Summative Evaluation Study of NanoDays 2014 Events: In the spring of 2014, the Nanoscale Informal Science Education Network (NISE Net) Public Impacts evaluation team conducted a summative study of NanoDays, a nationwide festival of educational programs about nanoscale science, engineering, and technology.
- The Public Impacts Mini-Exhibition Study Year 8 Summative evaluation focused on the reach of the Nano mini-exhibition and whether or not the exhibit was successful in providing visitors with an engaging and educational experience. This study also analyzed the mini-exhibition’s success in different contexts and for different types of audiences, including Hispanic visitors and visitors with disabilities, and whether or not it was a catalyst for new public programming around nano.
- A Year 6 Network Communication Study was conducted to learn about how the Network's primary communication components (NanoDays, face-to-face meetings, the regional hub structure, and the nisenet.org website) are being used by actively involved partners.
- The Exhibits and Programs Year 5 Summative Evaluation studied the impact of a set of NISE Net exhibits and programs on museum visitors.
- The Year 5 Delivery and Public Reach Study examined nano educational activities that NISE Net partners are doing and estimated the public reach of the Network in 2010.
Review of NISE Network Evaluation Findings: Years 1-5 is a review of the findings from over 240 evaluation reports from the first five years of the Network. The review is divided into six chapters:
- Connecting ISE Professionals with Nano Informal Science Education
- Connecting the University-Affiliated Individuals with Nano Informal Science Education
- Engaging the Public in Learning about Nano through NISE Network Educational Products
- Engaging the Public with Societal and Ethical Implications Content through NISE Network Products
- Making the Unfamiliar Interesting and Relevant for Museum Visitors
- Reaching Public Audiences
The NISE Net Evaluation Team has also conducted front-end and formative studies in order to guide the development of most NISE Net products and practices.
During the early stages of a project, front-end evaluation is used to learn about visitors’ familiarity with a topic, their interests and feelings, and what understandings they bring to an experience. Examples of front-end studies for NISE Net have included reviews of existing projects about nanoscience and a study investigating public awareness, interest, knowledge, and attitudes related to nanotechnology. Links to these studies are below:
- Nanotechnology and the Public Year 1 Front-End Evaluation is a 2005 literature review that summarized findings from secondary sources that describe public awareness, interest, knowledge and attitudes related to nanotechnology.
- Compilation of Nanoscale Exhibit Projects Year 1 Front-End Evaluation is a 2005 literature review of existing Nanoscale Communication Projects.
During the design and development stage of a project, formative evaluation is used to help achieve desired goals; formative evaluation involves iterative testing and modification of prototype exhibits, materials, and programs.
In the first five years of the Network, all formative evaluations were conducted by members of the Evaluation Team and included several rounds of testing of exhibit prototypes, program work such as forums and stage presentations, along with interactive media products. In the second five years of the Network, formative evaluations were conducted by NISE Net practitioners through a Team-Based Inquiry process.
Below are examples of formative studies that were conducted by the NISE Net Evaluation group in the first five years of the network.
- Who Decides Forum is a formative evaluation that looked at ways to improve visitor enjoyment, engagement and perceived learning at a discussion forum that was held at five institutions.
- Intro to Nanotechnology is a formative evaluation that explored the effectiveness of an initial set of NISE Net prototype exhibits in 2006.
- Exploring Measurement-Stretchability is a formative evaluation that looked at children and adults’ engagement and comprehension of a nano activity focused on the nanoscale.
- Treating Tumors with Gold is a formative evaluation that gathered visitors’ feedback to a stage presentation on research using nanoshells and infrared light in cancer therapy.
More NISE Net evaluation reports are available within the following section of the website. Where possible, reports are also included as resources for each of the related products.
Team-Based Inquiry (TBI) is a practical approach to empowering education professionals to get the data they need, when they need it, to improve their products and practices and, ultimately, more effectively engage public and professional audiences. The TBI process involves an ongoing cycle of inquiry: question, investigate, reflect, and improve.
The Team-Based Inquiry Guide explains each step of the TBI process and features ways TBI is used in the NISE Network to improve educational experiences and professional practice.
NISE Net Guidelines for Collecting and Handling Data outlines general guidelines for collecting and handling TBI data. The guide was created for the NISE Network's Institutional Review Board (IRB) protocol as a part of the Human Subjects Research training for NISE Net partners.
The Team-Based Inquiry Training Videos were designed to support museum practitioners in learning about and using TBI in their own institutions by describing this process and providing examples of how a team might work through the different steps.
Over the course of the project, the evaluation team has included staff members from the following organizations:
- Museum of Science, Boston
- Science Museum of Minnesota
- Oregon Museum of Science and Industry
- Inverness Research Associates
- Multimedia Research
In addition, the summative evaluation work for years 5-10 has been overseen by an external Committee of Visitors. This committee currently includes the following members:
- Frances Lawrenz
- Bruce Lewenstein
- Saul Rockman
Carol Weiss was also a member of the Committee of Visitors for several years.
NISE Network's online surveys are powered by SurveyGizmo.
The NISE Network is conducting several multi-year research projects looking at different aspects of the Network that have the potential to inform the field of informal science education. Whereas the evaluation studies conducted by the NISE Network measure progress toward NISE Net goals, the research studies are designed to generate findings that will inform and advance the broader field of informal science education. Below is a description of these projects. For more details, contact members of the research team.
Research Reports on the website
Research reports summarize the results of research studies conducted on the impacts of informal science education efforts on both public audiences and museum professionals.
- Nano and Society Case Study of a Research-to-Practice Partnership between University Scientists and Museum Professionals - 2014
- Partnerships in the Nanoscale Informal Science Education Network (NISE Net): A study of partnerships between university scientists and museum professionals - 2015
- NISE Net Research on How Visitors Find and Discuss Relevance in the Nano Exhibition - 2015
- NISE Network Research - Nano Online: Tracking NISE Net’s Digital Footprint - 2015
- Additional reports will be added to the website as soon as they are completed.
This project examines the nature and impacts of the development of partnerships between informal science institutions and practicing scientists. We are interested in learning how museum-scientist partnerships create useful and meaningful representations of complex and cutting-edge scientific and socioscientific concepts; how these representations underpin the development of informal education materials, experiences and approaches that are used by museum exhibit designers and program staff; and how new products and practices resulting from museum-scientist partnerships are used in supporting and expanding the NISE Network. A case study follows the formation of the Nano & Society partnership between museum educators and scientists through its inception to the four regional workshops given on the topic. Another study, the network analysis of survey data, provides insightful information about connections between organizations within the NISE Network.
- Timeline: 2011-2015
- Ways to get involved: This research project is actively seeking partners within the NISE Network to collaborate on this work who currently are or have historically been working collaboratively to produce representations that become part of exhibits, materials, designed experiences, and other products in the network. There are a variety of ways for you to help us: (a) Let us know if you are available for an interview; (b) Allow us to arrange a focus group with you as a participant; (c) If you are forming a new museum-scientist partnership to develop a product, allow us to attend your working sessions.
For more information:
- Tina Stanford, SRI, firstname.lastname@example.org
- Research Reports:
This study examines whether and how organizations affiliated with NISE Net experience widespread and long-lasting change in their practices. Previous studies from formal education and other fields have shown that networks can serve as a powerful impetus for change as they enable the introduction of new practices and ideas into an organization through frequent interactions with other similar organizations. This study looks at partners and explores the conditions that facilitate or impede the adoption of NISE Net-related knowledge and practices within these organizations.
- Timeline: 2011-2015
- For more information:
This project focuses on exploring the different ways NISE Net can engage visitors in learning and decision making about nano. Currently, this project is studying how visitors use, interact with, and talk about the different exhibit components within the Nano mini-exhibition to learn about the different areas of the NISE Network content map. In particular, the team will research how visitors’ understandings of relevance, or connection to their lives, develops within the exhibition.
- Timeline: 2011-2015
- For more information:
- Research Reports
This project examines media coverage and online discourses about nanotechnology, with a particular focus on NISE Network-related work, through tracking media content, blog posts, and discussions on various social media platforms (e.g., Facebook and Twitter). We use combinations of commercial software packages, including Lexis Nexis, Vantage Point, and Crimson Hexagon, to track large amounts of communication offline and online in a real time fashion.
First, we track the volume of coverage in traditional media channels and discussions on various online platforms. Second, we monitor genuine and spontaneous responses on nanotechnology-related topics among concerned publics in online environments. This project tracks sentiment, expressed with different levels of certitude, and thematic content such as risk-benefit assessment, of nanotechnology-related topics, especially NISE Network activities. Third, we examine the relationship between NISE Network events across different regional sites and the traffic of relevant discussions on both traditional and online media channels. For instance, this project will track online traffic surrounding particular sites before and after events such as NanoDays or other NISE Network coordinated activities, and explore the relationship between the tracked traffic and the corresponding geographical origins of online discussion (such as Tweets).
- Timeline: 2013-2015
- For more information:
- Research Reports: