3,197
edits
Changes
→Efforts to Improve & Streamline the Review Process
==Efforts to Improve & Streamline the Review Process==
==Efforts to Improve & Streamline the Review Process==
===2014-15 Standardization Efforts===
===="Audit" Approach and First Steps====
At the beginning of 2014, the [[Operational Effectiveness Committee|Structural Improvements Committee (SIC)]] (as it was named at the time) investigated the possibility of unifying the process for Article 4 reviews, so that each review, regardless of organization, followed a predictable path. The proposal from the chair included the adoption of an "audit" approach to reviews:
<blockquote>The Chair discussed elements of different types of audits (outcome, structural and process element audits), and how they may be used to assist the overall effectiveness of the organization. The SIC considered potential elements of an audit or review mechanism, including the methods and tools, and qualitative and quantitative metrics. The Chair noted that the outputs could be fed into the work of other Board committees, such as the Audit Committee or the Risk Committee. The members of the SIC discussed whether the proposed approach would overlap with the overall organizational assessment that is part of the ATRT review. The SIC also discussed the implications of using the term "audit" to identify the structural review, and discussed the importance of ensuring that any review (or "audit") meets the requirements in the Bylaws.<ref name="febsic">[https://www.icann.org/resources/board-material/minutes-bsic-2014-02-06-en Meeting Minutes, Structural Improvements Committee], February 6, 2014</ref> </blockquote>
Committee members expressed some concern about the term "audit," which in both definition and practice is distinct from a "review,"<ref>''See e.g.'' the Government Accountability Office's "[https://www.gao.gov/products/113596 Audit and Evaluation: Is there a difference?]" - October 6, 1980</ref> and noted that any review/audit process must comply with the ICANN Bylaws. Committee members also noted community concerns about the number of reviews underway.<ref name="febsic" />
====Impact on GNSO2====
The introduction of the "audit" concept coincided with the initiation of the [[Second GNSO Organizational Review]], and appears to have influenced the scope described in the RFP for that review. Notably, there is no indication that the ICANN Board reviewed or approved a Terms of Reference document regarding GNSO2.<ref>[https://www.icann.org/resources/reviews/org/gnso GNSO Organizational Review Dashboard] (Note that GNSO2 was "initiated" and a "Terms of Reference approved" by the board, with no citation to a source document or to meeting minutes.)</ref> In March 2014, both the "audit" concept and GNSO2 were on the agenda at the SIC meeting.<ref name="marchsic">[https://www.icann.org/resources/board-material/minutes-bsic-2014-03-21-en Meeting Minutes, Structural Improvement Committee], March 21, 2014</ref> Although the term "audit" had disappeared, the discussion of a "review accountability framework" included similar categories to those discussed in February: "process reviews, process element reviews, outcome reviews, and structure reviews."<ref name="marchsic" /> ICANN staff was directed to:
<blockquote>provide a comprehensive view of how the different review mechanisms correlate and what assurances they provide; identify standards and criteria used by the previous review; and identify opportunities to streamline review mechanisms and processes for increased efficiency and effectiveness.<ref name="marchsic" /></blockquote>
In the discussion of GNSO2, meanwhile, the presentation emphasized the intentionally limited scope and process of the review, echoing comments made in concurrent presentations to the GNSO and the public.<ref name="marchsic" /> In particular, GNSO2 would ignore the first question of Article 4.4 - whether the GNSO served a continuing purpose in the ICANN structure.<ref>For more background, see ICANNWiki's article on the [[Second GNSO Organizational Review]]</ref>
====Shift from "Audit" to "Unified Approach"====
At the June meeting of the SIC, a representative from [[Moss Adams]] provided a report on their engagement to conduct an inventory and mapping of reviews with ICANN and their existing processes, for the following purposes: "(a) to identify how the different review mechanisms correlate and what assurances they provide; (b) to identify standards and criteria used by prior reviews; and (c) to identify gaps and/or redundancies between the reviews."<ref>[https://www.icann.org/resources/board-material/minutes-bsic-2014-06-20-en Meeting Minutes, Structural Improvement Committee], June 20, 2014</ref> While efficiency concerns were an aspect of the Moss Adams study, the primary thrust was now compliance with the requirements of the bylaws, and establishing baseline operational standards for past reviews.
At the SIC meeting in October 2014, staff presented the outcomes of the study:
<blockquote>Staff provided the SIC with an update on the inventory and mapping of the reviews within ICANN performed by Moss Adams. The purpose of the report is to identify gaps, overlaps and dependencies within the structural reviews and Affirmation of Commitments reviews. A few recommendations were provided, including development of overall review management procedures and centralized tracking, as well as development of primers for independent examiners to allow for greater consistency.<ref name="octsic">[https://www.icann.org/resources/board-material/minutes-bsic-2014-10-12-en Meeting Minutes, Structural Improvements Committee], October 12, 2014</ref> Staff was instructed to generate topic areas for discussion, as well as a plan for productively engaging those topic areas at the next SIC meeting.<ref name="octsic" />
In February 2015, staff proposed three "buckets" for discussion and potential improvements to the SIC: "process improvements, oversight and coordination, and a more strategic look at the "structures" within ICANN as a whole."<ref name="feb15sic">[https://www.icann.org/resources/board-material/minutes-bsic-2015-02-06-en Meeting Minutes, Structural Improvement Committee], February 6, 2015</ref> Staff was instructed to begin work on policy and procedures documents, as well as a tracking model, with an eye toward standardization of process across Article 4 reviews.<ref name="feb15sic" /> Also in February, at [[ICANN 52]], the schedule for both the "Affirmation of Commitments" reviews and organizational reviews was discussed, along with the ongoing efforts to bring standardization to the review process.<ref>[https://archive.icann.org/meetings/singapore2015/en/schedule/mon-aoc-org-reviews.html AoC and Organizational Reviews: Supporting ICANN Accountability], ICANN 52, February 9, 2015</ref>
====Process Improvement Options at ICANN 53====
After a particularly full agenda precluded discussion in April, the committee next addressed the review framework at [[ICANN 53]] in Buenos Aires. At the SIC meeting during ICANN 53, staff proposed several areas for potential process improvements:
* Standard Review Process and Methodology
* Relevance to Each Organization
* Adoption and Application of Standards
* Increased Effectiveness and Impact
* Creating Alternate Process for Strategic Reviews<ref name="53sic">[https://www.icann.org/resources/board-material/minutes-bsic-2015-06-19-en Meeting Minutes, Structural Improvements Committee], June 19, 2015</ref> The other buckets - oversight and holistic approaches to ICANN's structural issues - were wrapped into the [[ICANN 52 Session]] on review efficiencies, and a public comment period was still open at the time of the meeting.<ref name="53sic" /> Public comments on issues surrounding reviews were wide-ranging, but it is notable that multiple commenters objected to the exclusion of structural considerations from the GNSO2 review.<ref>[https://www.icann.org/en/system/files/files/report-comments-aoc-org-reviews-05aug15-en.pdf Staff Report on Public Comment Proceeding - AoC and Org Reviews], August 5, 2015</ref>
====Proceeses Proposed, Attention Wanes====
In September 2015, the [[Organizational Effectiveness Committee#History|newly-renamed]] Organizational Effectiveness Committee acknowledged that lessons learned from the last series of reviews could be refined into process improvements applicable to all reviews within ICANN.<ref name="septoec">[https://www.icann.org/resources/board-material/minutes-oec-2015-09-28-en Meeting Minutes, Organizational Effectiveness Committee], September 28, 2015</ref> The discussion and commitment to engage in such refinemnent appeared to mark the end of work in regards to the "process improvements" bucket.
With regard to a "review framework," the October 2015 meeting of the OEC resulted in the following two-phase plan:
<blockquote> Phase 1: Systematization of Reviews, based on current mandates
<ol>
<li> Document the process of conducting Reviews, based on current practices and lessons learned from recent Reviews</li>
<li> Incorporate applicable and relevant standards derived from industry best practices and standards, such as Project Management (Project Management Institute) and Organizational Excellence (EFQM Excellence Model)</li>
<li>Socialize and implement systemization</li>
Phase 2: Future of Reviews in the post-transition ICANN
<li>In alignment with the direction of CCWG-Accountability and other related proposals impacting Reviews, identify key questions for consideration of the future of Reviews as a significant accountability mechanism</li>
<li>OEC discussion</li>
<li>Develop next steps as appropriate<ref name="octoec">[https://www.icann.org/resources/board-material/minutes-oec-2015-10-17-en Meeting Minutes, Organizational Effectiveness Committee], October 17, 2015</ref></li>
</ol>
===2019 Operating Standards Updates===
Both the ICANN Board and ICANN staff have recently been engaged in efforts to improve the review process for both specific and organizational reviews. In 2019, the Board issued new Operating Standards for Specific Reviews<ref>[https://www.icann.org/en/blogs/details/operating-standards-guiding-icanns-specific-reviews-8-7-2019-en ICANN.org Blog - Operating Standards: Guiding ICANN's Specific Reviews], July 8, 2019</ref> <ref>[https://www.icann.org/en/system/files/files/operating-standards-specific-reviews-23jun19-en.pdf ICANN Operating Standards - Specific Reviews], June 23, 2019 (PDF)</ref> In addition, ICANN staff drafted a process proposal for streamlining organizational reviews in April 2019.<ref>[https://www.icann.org/public-comments/streamlining-org-reviews-proposal-2019-04-30-en ICANN.org - Public Comment Archive], Process Proposal for Streamlining Organization Reviews, April 30, 2019</ref> Public comments on the proposal addressed a much broader range of challenges and difficulties than the proposed streamlining measures.<ref>[https://www.icann.org/en/system/files/files/report-comments-streamlining-org-reviews-proposal-30jul19-en.pdf Staff Report on Public Comment Process], July 30, 2019</ref>
Both the ICANN Board and ICANN staff have recently been engaged in efforts to improve the review process for both specific and organizational reviews. In 2019, the Board issued new Operating Standards for Specific Reviews<ref>[https://www.icann.org/en/blogs/details/operating-standards-guiding-icanns-specific-reviews-8-7-2019-en ICANN.org Blog - Operating Standards: Guiding ICANN's Specific Reviews], July 8, 2019</ref> <ref>[https://www.icann.org/en/system/files/files/operating-standards-specific-reviews-23jun19-en.pdf ICANN Operating Standards - Specific Reviews], June 23, 2019 (PDF)</ref> In addition, ICANN staff drafted a process proposal for streamlining organizational reviews in April 2019.<ref>[https://www.icann.org/public-comments/streamlining-org-reviews-proposal-2019-04-30-en ICANN.org - Public Comment Archive], Process Proposal for Streamlining Organization Reviews, April 30, 2019</ref> Public comments on the proposal addressed a much broader range of challenges and difficulties than the proposed streamlining measures.<ref>[https://www.icann.org/en/system/files/files/report-comments-streamlining-org-reviews-proposal-30jul19-en.pdf Staff Report on Public Comment Process], July 30, 2019</ref>