How we investigated State Integrity

Introduction

The State Integrity Investigation assesses the existence, effectiveness, and accessibility (i.e. citizen access) of key governance and anti-corruption mechanisms through a qualitative and indicator-based research process.

A first round of inquiry was conducted in 2011 and published in 2012, assessing the state of integrity in all 50 states. In 2014, Global Integrity again partnered with the Center for Public Integrity to embark on a second round of research. The indicators diagnose the strengths and weaknesses of the institutional safeguards applied against corruption in each state – mechanisms of openness, transparency, and accountability – rather than corruption itself. The indicators comprise 13 sections divided into 92 de jure indicators, 153 de facto indicators. Categories assessed include:

• Section 1: Public Access to Information
• Section 2: Political Finance
• Section 3: Electoral Oversight
• Section 4: Executive Accountability
• Section 5: Legislative Accountability
• Section 6: Judicial Accountability
• Section 7: State Budget Processes
• Section 8: State Civil Service Management
• Section 9: Procurement
• Section 10: Internal Auditing
• Section 11: Lobbying Disclosure
• Section 12: Ethics Enforcement Agencies
• Section 13: State Pension Fund Management

To identify the project’s indicators, staff from Global Integrity and the Center for Public Integrity contacted nearly 100 state-level organizations and experts working in the areas of good government and public sector reform around the country during 2011. We asked what issues mattered most in their state when it came to the risk of significant corruption occurring in the public sector. The outcome was a list of more than 300 questions, rooted in the inner workings of state government operations. Based on these discussions, Global Integrity and the Center for Public Integrity also selected and included relevant indicators from previous projects in the overall indicator set.

In order to further strengthen the overall set of questions — and building on the lessons learned from the project’s first round — Global Integrity reduced the overall number of indicators while maintaining the project’s scope. This decision was taken in light of feedback from various stakeholders, with an aim to assessing the most salient corruption risks in each state. GI also revamped and refined the scoring conditions of select indicators in order to increase the reliability and validity of the assessment. As part of this process, the investigation dropped two sections from the previous round of research: “State Insurance Commissions” and “Redistricting,” while adding a section on “Electoral Oversight.” As a result, the total number of indicators dropped from 330 to 245.

Evidence-based expert assessment methodology

Typical Global Integrity assessments provide detailed data and reporting on the mechanisms in place to prevent abuses of power and promote public integrity at the national, sub-national and at sector levels, both in the U.S. and abroad. Using a blend of social science and journalism, local teams of independent researchers and journalists report on the de jure as well as de facto reality of corruption and governance. Measuring the scope of the legal framework and what occurs in practice is key in our effort to produce actionable governance data that help governments, citizens and civil society understand and evaluate the status quo and identify intervention points for possible reform efforts.

Global Integrity’s evidence-based expert assessments require researchers (or journalists, in the case of the State Integrity Investigation) to compile and document evidence to inform and support their score choices for each of the indicators. Rather than relying on experiences or pre-existing perceptions by experts, the strength of Global Integrity’s methodology is that it requires a variety of information sources to be reviewed and documented (including legal and scholarly reviews, the text of government laws and procedures, interviews with experts, and reviews of media stories) to substantiate the score choice and triangulate the data points identified.

Various factors can affect the interpretation of a particular indicator and the score assigned to it. To minimize this effect and to maximize inter-coder reliability, Global Integrity provides researchers and peer reviewers with scoring criteria for every indicator. The scoring criteria anchor each indicator to a predefined set of conditions. In essence, the scoring criteria guide the researcher and the peer reviewer by suggesting, “If you see X on the ground, score this indicator in the following way.”

For “in law” indicators, scoring criteria are provided for “Yes (100)” “Moderate (50)”and “No (0)” responses. For “in practice” indicators, scoring criteria are provided for 100, 50 and 0 – but researchers also have the option to score 75 or 25, whenever the higher or lower defined criteria don’t accurately represent the researcher’s findings.

Researchers must provide a fact-based explanatory comment to substantiate the score chosen for each indicator. They must also provide at least three references to substantiate each comment for any “in practice” indicator. Evidence can take on the form of interviews conducted with knowledgeable domain experts, a website link to information provided by government or civil society, a relevant report produced within the period of study, published news articles, or a widely acknowledged and regularly-read blog produced by a subject matter expert.

In the case of de jure indicators, the laws or regulations and a reference to relevant articles or legislative/regulatory sections serve as evidence. In some cases, more than just one law is needed as a valid reference, and in those cases any number of laws or other aspects of the legal framework (including court decisions) may serve as evidence.

Global Integrity staffers rigorously verify sources and perform fact-checking and quality control together with the researcher prior to declaring the research as final. After completion of the draft research, the indicators follow a double-blind peer review process, in which a peer reviewer (a similarly qualified in-state domain expert) carefully scrutinizes and amends, supplements and enhances the research. The double-blind nature of the peer review process guarantees feedback free of considerations associated with who collected the data and scored the indicators, and avoids a peer-influenced consensus. Until the public release of the indicators, the researchers and peer reviewers are – by default – unaware of the identities of other members of the team.

Along with the scoring criteria, researchers and peer reviewers are also provided with a series of guidelines and definitions they must adhere to when performing their research and coding or reviewing the data. These guidelines also provide guidance about appropriate sources and about how to score complex indicators in certain scenarios. For example, we ask about political parties and individual candidates in one question – offering scoring options to intelligently and consistently score cases in which only one element is applicable.

Data Aggregation
For the purpose of producing a state’s aggregate report card, a simple aggregation method was used. As described above, the reporter assigned a score to each indicator. Each score was then averaged within its parent subcategory, which produced a subcategory score. The subcategory score was in turn averaged with the other subcategory scores into a parent category score. Category scores were averaged to produce an overall state score. The State Integrity Index groups states into the following 11 performance “tiers” according to a state’s overall aggregated score:

• 90 and above: A
• 87–89:B+
• 84–86: B
• 80–83:B-
• 77–79:C+
• 74–76: C
• 70–73:C-
• 67–69:D+
• 64–66: D
• 60–63:D-
• 59 and below: F

Because some aspects of governance and anti-corruption mechanisms are harder to measure definitively, some categories require a more complex matrix of indicators than others. Thus, the categories are equally valued, even if some categories are derived from a more lengthy series of indicators than others. Similarly, the subcategories are equally valued within their parent category.

Due to the difference in the number, content, and scoring of indicators between the first and second round of research, making comparisons between individual indicator scores from one round to the next can be misleading. Despite this, notwithstanding the methodological changes made, Global Integrity has retained the comparability of section scores. However, users of the data are encouraged to note that neither aggregate quantitative scores nor individual indicator scores capture the scope of an indicator answer or full scorecard for a given state. In order to grasp what a given score signifies at the state level, please refer to the comments and sources for each indicator.

Period of Study
Researchers were asked to research the effectiveness of laws — based on the evidence — for the period of time covering January 2013 through March 2015.

The cut-off date for de-jure indicators has been June 30, 2015. All bills that have passed the legislative and executive branch approval processes and are slated to become effective on or before January 1, 2016 have been included as valid legal instruments into the research.

State Teams
The Center for Public Integrity hired experienced journalists in each state to serve as researchers. Following Global Integrity’s traditional policy, none of the researchers has worked directly for state government during the past three years. In each state, Global Integrity also contracted a peer reviewer with a similar background to review and fact-check the initial research where needed. The teams were coordinated from Washington DC via the Internet and phone.

Enforcement Gap

The State Integrity Index also measures what we term the “enforcement gap” for each state. The enforcement gap refers to the difference between the state’s legal framework for good governance and anti-corruption and the actual implementation and enforcement of that same legal framework. We generated an enforcement gap at the state score level as opposed to the category and/or subcategory levels.

The enforcement gap is created by first generating a “legal framework score” and then an “implementation score” for each state. Once the legal framework and actual implementation scores have been generated, we simply subtracted the implementation score from the legal score to generate the enforcement gap for a given state.

Contact us

The State Integrity Investigation invites local and national media to both report on our findings and use Investigation data to construct additional stories. We can provide spreadsheets and other material to help journalists shine light on their state government’s openness and accountability.

Media contact

For media inquiries or to arrange interviews with the State Integrity Investigation’s researchers or journalists, please contact:

William Gray
Center for Public Integrity
wgray@publicintegrity.org or 202-481-1232