Program Monitoring: Assess whether the campaign is achieving milestones for implementation and operations.

Strategic Monitoring: Assess whether the campaign is on track to achieve desired outcomes by using strategic measures and metrics.

Operational Monitoring: Assess whether campaign operations are efficient and effective to achieve operational outcomes.

• Establish a modeling structure for the current and projected impacts of the PHE.
  • Modeling may inform campaign components such as community activation and partner opportunities, media options and message development, and how actions taken by the campaign might influence the PHE.
• Identify data analysts with modeling expertise at the outset of the planning process to provide the campaign with findings to support campaign decisions.
• Regularly assess both the actual progression and modeled forecasts of the emergency’s effect on populations.

• Establish an evidence-based modeling program that leverages both actual values and predictive estimates to understand the extent of the emergency and how it changes over time.
  • Incorporate data on the number of cases, fatalities, people requiring treatment or hospitalization, population(s) to target (e.g., for blood or tissue donations, vaccination, or other intervention), and results from the campaign actions (e.g., number of signups to donate or get vaccinated, number of website visits).
  • Model and simulate to help predict where resources should be targeted.
  • Investigate the effectiveness of the campaign to increase awareness, intent, and action.
  • Visualize the key metrics in a dashboard to make the data easier to understand and more widely available to stakeholders.
• Weigh multiple (and sometimes conflicting) factors, including resources to make campaign decisions.
  • Depending on the complexity and scope, this may be accomplished with a simple multi-criteria analysis framework or a more advanced optimization.
  • Optimization may be required, for example, to identify the best locations to supplement capacity or where to invest in the campaign.

• Develop a modeling decision framework.
  • Define the scope, including regions (e.g., state, metropolitan statistical area [MSA], county) and communities of interest.
  • Define decisions to be made during the campaign.
  • Identify key metrics to support decisions.
  • Identify a methodology to evaluate decisions.
• Determine whether the campaign team has sufficient expertise and access to existing modeling resources and capabilities within the organization (e.g., experience in simulation modeling, data analytics, and visualization as well as coding experience in modeling and dashboard tools).
  • Determine any gaps in expertise and supplement staffing with external resources that can be procured early in campaign planning to inform decisions.
• Conduct, at minimum, weekly analysis to see changes in models and predictions of how things may change with the emergency.
• Provide weekly updates that include predictive modeling results, essential information, and insights to the CMT and the oversight team.

It is important to monitor the online landscape by using social listening tactics and conducting ongoing sentiment analyses of narratives shared across media channels. These activities can help inform communications and address public concerns before they are amplified and before the knowledge gap is filled by misinformation and/or disinformation. Monitoring evolving science and public sentiment may also help identify credible influencers or opinion shapers who may be approached to serve as trusted messengers for the campaign.

Key Decisions and Activities to Monitor the Science and Public Sentiment

• Develop a regular schedule to monitor social media channels for relevant topics and how they are manipulated.
  • The schedule will provide situational awareness of current topics and themes in public discourse, the amplification of misinformation and/or disinformation, and public sentiment on the PHE.
• Monitor published and pre-published medical journals released with the most up-to- date information on the PHE, including treatment and prevention.
• Analyze and report social media sentiment for valuable information to review and revise campaign messaging and strategy
• Studying shifts and intensity in online conversations may convey the public as a skewed or incomplete understanding about the underlying health issues or how to interpret and apply scientific and clinical guidance. Review how populations translate information and how they discuss it (e.g., assigning positive or negative attributes to currently available information; expressing changed emotional reactions to the PHE, the government, or the clinical community).
• Establish a regular analysis and reporting schedule (at minimum weekly) and integrate the schedule into the overall data reporting structure.
• Share findings with the clinical and scientific advisory body.
  • Clinicians and scientists can help determine if a mitigation intervention is warranted.
  • Enhanced public education and countermeasures can be devised to promote a scientifically accurate public discourse.
• Share findings with the communication team.
  • Potential shifts in the messaging strategy can be identified and implemented.
• Identify information that may be a potential risk to the overall response strategy.
• Identify potential influencers and opinion shapers that could be partners for community engagement or message amplification.

For a campaign to be successful, it is necessary to employ structured management principles with key milestones tracked and progress updates delivered to the oversight team. Milestones are the crucial interim actions or steps that must be achieved reach full campaign implementation. These activities monitor the campaign at the operational level.

Example milestones include:

• Establishment of campaign goals and objectives
• Asset testing and refinement
• Campaign launch
• Initial evaluation, metrics review, and campaign refinement
• Integration of scientific (clinical and research) findings into campaign decision-making
• New campaign outreach based on interim learnings
• Reports on achievements on strategic or operational measures and metrics
• Reports on campaign management process evaluation (Evaluate the Campaign Management Process)
• Reports on campaign outcome evaluation (Evaluate the Campaign Strategic Outcomes)

Campaign leaders must craft and implement well-founded measures and review findings systematically to discover whether the campaign is on track to achieve its desired outcomes. The cadence for measuring and reporting measure status will be driven by the oversight team and by data availability. These activities monitor the campaign at the strategic level.

Separately or in collaboration with the oversight team, the campaign management team (CMT) will develop and use operational measures and metrics regularly to inform whether campaign activities perform efficiently and effectively. These activities monitor the campaign at the operational level.

Campaign outcomes are used to develop measures to observe campaign community education and behavioral activation effectiveness (Define Desired Outcomes). An outcome evaluation shows the degree to which the campaign is influencing the target population’s behaviors. It tells whether the program is effective in meeting its outcomes and objectives; whereas a process evaluation indicates how the campaign activities contributed to reaching overall program outcomes. The outcome evaluation is only effective after the campaign has contacted the target population and is able to collect enough data over time.

The following are methods for evaluating strategic outcomes (Monitor Campaign Strategic Measures and Metrics):

• Modeling and simulation can be used to compare different strategies’ effectiveness to influence optimal results for people and communities.
• Data analysis, including exploratory and predictive modeling, can be used to investigate the effectiveness of the campaign efforts on the measured outcome.

Campaign management process evaluation measures how well the campaign is working. It includes analyzing metrics for campaign activities and accomplishments by describing measurable outputs (e.g., what was done, how much was done). It also measures whether the campaign is accessible and acceptable to its target population. This type of evaluation provides an early warning for any problems that may occur and allows the CMT to monitor how well the campaign plans and activities are working.

A process evaluation includes data, measures, metrics, and outputs from campaign strategy activities, such as:

• Website metrics and telemetry
• Paid media (e.g., radio, print, television), out of home display, and click-based media (e.g., paid search) metrics
• Social media measures and metrics
• Social listening metrics
• Community partner measures and metrics
• Community activities
• Audience awareness or knowledge
• Assets developed and deployed