Protocol 2.4: Evaluate evidence reliability

The IPC does not involve any form of primary data collection, but rather uses available evidence in its analysis and undergoes a comprehensive assessment of all available evidence based on established criteria to assign reliability scores (R). Evidence on both outcome indicators (such as GAM based on WHZ, and GAM based on MUAC) as well as contributing factors (e.g. evidence diseases, feeding practices, and water and sanitation) should be evaluated, and an R should be assigned for each piece of evidence.

Evidence used in the IPC can have a reliability score of R2= reliable or R1= somewhat reliable. R1 is further divided into two scores: R1+ and R1-. R1+ refers to evidence that has good time relevance but is limited in terms of the soundness of the method or indicator used; R1- refers to evidence that has limited time-relevance but is considered good in terms of the method and indicator used. Outcome evidence that is limited both in terms of soundness of method and time relevance cannot be used in the IPC Acute Malnutrition classification. The assessment of reliability is not based on a statistically rigorous assessment, but rather on a general assessment of the soundness of methods of data collection and indicators used (M) and the time relevance of the evidence to current or projected analysis (T).

The Reliability Score Table for Acute Malnutrition Evidence (shown in Table 38) presents the general criteria for assessing reliability scores and provides more specific guidance on the assessment of the M and T for Acute Malnutrition evidence:

  • Part A presents the combination of M and T that underpins the different reliability scores. Evidence is reliable when: it is based on a standardized indicator; the method used to collect the indicator is robust; and it depicts the current conditions. If the evidence is based on a non-standardized indicator (e.g. GAM based on MUAC), is yielded from a reasonable but less rigorous method (e.g. one with limited representativeness), or is based on inference (e.g. recent or historical evidence), it can be at most R1. Reasonable evidence that scores less than R1 can only be used in IPC Acute Malnutrition classification under special conditions – e.g. R0 evidence collected from areas with limited or no humanitarian access.
  • Part B presents the general working definition of good and limited M and T as well as specific guidance for assessment of reliability of evidence on indicators included in the Reference Table.

 

 

Considerations:

  • Surveys representative at the unit of analysis:  Surveys refer to collection of data from a specific population at a single point in time on nutrition outcomes and/or contributing factors. They are typically carried out on a subset of the population of interest (i.e. sample), and the results from the sample are then applied to the survey population. Samples from the survey populations are typically selected using simple, systematic or cluster sampling methods. Surveys should be designed to be representative at the IPC unit of analysis. The size of the sample will vary from survey to survey, and should be calculated separately for each survey based on a set of parameters such as expected prevalence, desired precision and design effect (for cluster surveys). Adequate sample sizes will ensure the precision of the survey estimates but not necessarily guarantee the validity (or accuracy) of the survey estimate. In order to assess the validity of anthropometric survey estimates, analysts must look at the Standardized Monitoring and Assessment of Relief and Transitions plausibility check results (see www.smartmethodology.org) for details. It should be noted that, in the case of Rapid Standardized Monitoring and Assessment of Relief and Transitions Surveys where samples are drawn from simple or systematic random sampling methods, a sample size of about 150 children would be adequate to get acceptable prevalence – e.g. about ±6.5% precision for an expected GAM prevalence of 20%, about ±3.5% precision for a GAM prevalence of 5%. For additional information, see http://smartmethodology.org/survey-planning-tools/smart-methodology
  • Season refers to “Acute Malnutrition season” and not food-security seasons such as pre-harvest, harvest or post-harvest. Different Acute Malnutrition seasons indicate the relative fluctuations in the levels of Acute Malnutrition – i.e. high/low levels of Acute Malnutrition. IPC AAcute Malnutrition Analyses should establish the Acute Malnutrition season in the area of analysis prior to the analysis. Acute Malnutrition seasons can be established based on the feeding centre admission data, nutrition survey data, surveillance data, etc.
  • Disaggregated survey data from a higher administrative level: Surveys should ideally be representative at the unit of analysis. However, under some specific circumstances (see below), GAM based on WHZ data from surveys designed to be representative at a higher administrative level than the unit of analysis can be re-analysed to obtain estimates for lower administrative units and used in the IPC analysis. The main deciding factor in the case of disaggregated survey data is the design effect. If the design effect of the GAM based on WHZ from the higher administrative-level survey is <1.3, this higher administrative-level estimate can be used for all lower administrative levels without disaggregating the data. If the design effect of the GAM based on WHZ obtained at the higher administrative level is between 1.3 and 1.7, the data should be disaggregated for lower administrative levels with ≥5 clusters and ≥100 observations, and the disaggregated estimates can be used based on the design effect:
    • If the design effect ≤1.7, use the point estimate.
    • If the design effect >1.7, use the lower bound of 95% confidence interval as the minimum phase (Note that minimum phase refers to the phase that an area would be classified as being in based on the lower bound of the confidence interval – i.e. the area would be at least in this phase). This is only an indicative phase. The final phase for the area should be decided by taking into account this indicative phase as well as the phases based on the point estimate and the upper CI and with convergence of evidence with the contributing factors.

It should be noted that if the design effect of the GAM based on WHZ obtained at the higher administrative level is >1.7, these survey data should not be disaggregated for lower administrative levels.

  • Sentinel sites are usually purposively selected sites using predefined criteria. Sentinel sites can be community- or facility-based, but only data from community-based sentinel sites can be used in the IPC. Prevalence estimates from sentinel sites should be obtained by combining data from all sites.
  • Screenings are rapid population-based assessments, typically conducted to obtain a quick idea of the situation. Although GAM based on MUAC data are typically collected through screening, GAM based on WHZ can also be collected during screening. The same sample size and coverage requirements apply regardless of the indicator.
  • Surveys from similar areas can be used to classify a given unit of analysis when evidence is unavailable from that unit of analysis. Estimates from similar areas can only be used if they are good in terms of time relevance and soundness of method. Before surveys from a similar area are used to classify an area, the similarity between the two areas must be established through documented evidence. Two areas may be considered similar if they follow the same livelihood, seasonality and ecological patterns and if surveys from both areas (same season) in the past indicated comparable estimates, etc. Additionally, there must be a documented analysis of contributing factors showing that there have been no significant changes in the context.
  • Unusual events refer to shocks that have an impact on Acute Malnutrition. There is a vast array of shocks, including but not limited to conflicts, disease outbreaks, displacement, droughts and floods. It should be noted that not all shocks would have an impact on Acute Malnutrition. For example, there has been no significant change in the Acute Malnutrition levels among the Syrian refugees even after years of conflict and displacement. Prior to the IPC Acute Malnutrition analysis, analysts need to review their contexts and determine the level of shocks and their likely impact on Acute Malnutrition levels.
  • Historical evidence can be used if it is good in terms of the soundness of method and is from the same season of analysis. Historical trend data must be converged with other contributing factors, and this analysis must be documented.