Archived—Industry Canada - Technology Partnerships Canada
h2 Early Adopters Program

Results Based Management and Accountability Framework (RMAF)

Archived

Table of Contents

1.0 Introduction

This document sets out a Results-based Management and Accountability Framework (RMAF) for the h2 Early Adopters (h2EA) program managed by Technology Partnerships Canada (TPC). The RMAF includes the following components:

This RMAF will enable TPC to assess the results of the activities, adjust operations, and continue to improve the h2EA program.

Throughout this document, the term "hydrogen technologies" is used to indicate "hydrogen technologies and hydrogen-related technologies."

2.0 Program Profile

The h2EA program is aimed at addressing the urgent need to accelerate market adoption of hydrogen technologies and other hydrogen-compatible technologies that facilitate the transition to a hydrogen economy and attract world-class talent and investment to Canada. Through Technology Partnerships Canada (TPC), support will be given to the establishment of integrated hydrogen complexes such as "Hydrogen Villages", "Hydrogen Highways" and other similar partnerships in locations across Canada.

This support will target multiple public- and private-sector partners to demonstrate these technologies and showcase Canadian capabilities. These partnerships will involve integrating hydrogen-compatible technologies and hydrogen production, storage and distribution technologies with fuel cell and related portable, stationary and mobile applications in a microcosm of a hydrogen economy. Results of this initiative will include real-world experience and expertise; early market adoption of hydrogen technologies and infrastructure needed to support their wide-spread use; and increased consumer and investor awareness.

2.1 Rationale

The overall objective of the United Nations' Framework Convention on Climate Change aims to stabilize greenhouse gas concentrations in the atmosphere. This will require substantial investments in technologies to fundamentally change, over the long term, how we produce and consume goods and services. The development, demonstration and deployment of climate change technologies is a long-term and expensive endeavour. Climate change technologies compete with well-established and embedded infrastructures. Technological innovation will play a key role in meeting Canada's international obligations. This is particularly true with respect to the promise of hydrogen not only as an energy source, but as the foundation of a new economy: the hydrogen economy.

The transition to a hydrogen economy offers a unique opportunity for Canada. The use of hydrogen (produced from alternative energy sources), fuel cells and other hydrogen technologies can significantly reduce emissions of greenhouse gases and other pollutants. This transition offers significant growth and investment opportunities for other sectors, and for communities and regions across Canada, to develop clusters of expertise and capabilities, attract talent and investment, and offer rewarding careers for young Canadians. Canada is well positioned to provide international leadership in the hydrogen sector.

While Canada is currently a world leader in the development of hydrogen technologies, its first mover advantage is not secure since other countries are making significant new investments toward a hydrogen economy and are attempting to attract Canadian grown technological leadership. In addition to this international competition, Canadian firms face many technology and marketplace hurdles which they cannot overcome alone. With respect to hydrogen technologies, significant headway must be achieved in reducing cost and improving their performance and safety. On the marketplace side, the most urgent need is to begin testing hydrogen technologies and infrastructure in real-world settings to assess and improve reliability and durability, to support efforts to further reduce costs, and to increase public, consumer and investor awareness and acceptance of hydrogen technologies. Implicit in a transition are necessary steps to an ultimate objective. Thus, emphasis is also required on technologies that are hydrogen-compatible.

Given the importance of social/behavioural change in the acceptance of new technologies, increased consumer and investor awareness and understanding of these new technologies and their uses are critical. There are significant technical, social, institutional and market barriers that cannot be overcome without government assistance.

Substantial progress has been and continues to be made through a variety of federal government initiatives in the areas of technology development and partnership activities, however greater collaboration and coordination is required to meet the next strategic objective - high profile, multiple partner market demonstration projects.

The h2EA program, managed by Technology Partnerships Canada, will support public and private-sector partnerships that showcase hydrogen complexes, for example "Hydrogen Villages" and "Hydrogen Highways" to demonstrate hydrogen technologies, illustrate their integration and accelerate market acceptance and take-up. Through demonstration projects, we will learn by doing: lessons learned from both successes and failures will advance the science and pace of technological development in this field. The historical experience has shown that these demonstration projects do not allow firms to rapidly realize main revenue streams, but rather help them to pursue further research and development on next generation technologies. Therefore, pursuant to the Treasury Board Policy on Transfer Payments, although contributions under the h2EA program are repayable, little or no repayments are expected.

Initiatives such as the h2EA program will allow Canada to stay at forefront of the transition to a hydrogen economy in order to maximize its benefits for Canadians.

2.2 Delivery Approach

The h2EA program will be implemented by Industry Canada through Technology Partnerships Canada (TPC) in conjunction with the regional development agencies and other federal departments and agencies and their respective programs. Memoranda of understanding may be developed between TPC and federal departments and agencies to support the implementation of this initiative.

During the period FY2003-04 to FY2007-08, funding will be made available by TPC to the Canadian Transportation Fuel Cell Alliance (CTFCA) to allow it to strengthen or extend its ability to meet the objectives of the h2EA program.

Regional development agencies will be invited to participate in the building of partnership coalitions, considering and monitoring the projects, and communicating the successes of regional initiatives.

Experts from appropriate Industry Canada sectors will be involved in the due diligence process.

Other federal departments and their respective programs will also be invited to participate, where their program objectives complement those of the h2EA program.

2.3 Primary Reach

In order for the h2EA program to be successful, it must "reach" several different types of organizations. These include organizations participating in hydrogen production, storage and distribution, the development of hydrogen and hydrogen-compatible technologies and users who will integrate hydrogen technologies into portable, stationary and mobile applications.

The primary clients and/or users of the h2EA program include:

In addition to some of the above groups, co-deliverers and participants include:

In addition to those listed above, there are a number of other stakeholders with interests in having a successful h2EA program operating in Canada. These include the Hydrogen and Fuel Cell Committee, National Research Council, NSERC, Canadian Transportation Fuel Cell Alliance, and other federal programs, which provide support and funding for research, development, and demonstration of hydrogen and hydrogen-compatible technologies, which relate to the results of the h2EA program.

2.4 Results

The Terms and Conditions for the h2EA program outline the following outcomes for the program:

These are reflected in the ultimate outcomes included in the logic model in the next section. The ultimate outcomes are:

In addition, the ultimate outcome of the h2EA program is to make a contribution to the objectives of Canada=s Climate Change Agenda.

2.5 Governance Structure

TPC will be responsible for design, delivery and administration of the h2EA program. The chart in the following exhibit illustrates the relationship between the key groups related to TPC and the h2EA program.

Figure 1 - h2EA program Organizational Chart

The roles of key groups involved in the h2EA program are shown in the table below.

Table 1 - Roles and Responsibilities of Key Groups
Role Responsibility
Program design TPC
Outreach / communication TPC
Program delivery TPC in collaboration with Regional Development Agencies
Monitoring and reporting TPC
Provision of advice Hydrogen Fuel Cell Committee

3.0 Logic Model

The logic model for the h2EA program is provided in Figure 3. For the purposes of the model, the following definitions were used:

As shown in Figure 2, TPC has varying degrees of control and influence over the success of the program.

Figure 2 - A Basic Results Logic

Note: Basic results logic as adapted from Exhibit 3.2 of the Guide for the Development of Results-based Management and Accountability Frameworks, TBS.

The basic results logic for the h2EA program initiatives should flow as follows:

The logic model for the h2EA program appears in Figure 3.

Figure 3 - h2 Early Adopters Program Logic Model
Objective: To work in partnership with hydrogen industry stakeholders to foster the demonstration and early introduction into the marketplace in Canada of technologies, such as fuel cells and those used to produce, store and distribute hydrogen, that support the transition to the hydrogen economy

4.0 Performance Measurement and Evaluation Strategies

On-going performance measurement allows for regular assessment of the progress made towards desired outputs and outcomes. Evaluations provide periodic reviews of a program, policy or initiative and its associated projects on a more rigorous and in-depth scale. These two approaches should combine to provide the information needed to manage for results. This RMAF proposes that the h2EA program will collect performance information according to two strategies:

As with the logic model, the relative control and influence of TPC are reflected here. For the activity and output measures, TPC has full control and is able to collect all of the associated measures. For example, TPC's fiscal management processes will provide the associated on-going financial measures. As the control of TPC turns into influence on partners/co-deliverers, the measurement strategy focuses on the collection of information which will describe the nature of these relationships. This information can then help maintain and improve these partnerships with other federal departments and funding recipients. Through a review of the content and frequency of communications with these key partner groups, the project leads can find the tone and tenor of the relationships, indicating the success of the h2EA program in developing these important partnerships.

When measuring the progress towards the intermediate outcomes of the program, where the influence of external factors begins to play a larger role, TPC will rely upon both their implementing departments / agencies and funding recipients to help gather and report on results.

The external influences are greatest on the ultimate outcomes and TPC exerts only limited influence. To reflect the level of external influence at this stage of the program, an evaluation will take place to report on the success of the program. This evaluation will summarize the

projects that have been completed according to expectations and standards. The program's metrics will be summarized, and will be contextualised within these external influences examining the broader communities' use of the infrastructure and the projects' contribution towards the final outcomes. The concept of learning by doing is fundamental to a demonstration program such as the h2EA program. This implies that the program, projects and other stakeholders learn from the successes and challenges encountered in implementing the demonstration projects. Courses corrections will need to be recognized and accepted in project monitoring and have been reflected in the ongoing performance measurement and evaluation strategies.

4.1 Ongoing Performance Measurement

Ongoing performance measurement provides a regular assessment of the performance of the program. The information provided by performance measurement helps to provide managers with the knowledge required in order to manage effectively. This strategy will identify the key pieces of information that need to be collected (in the form of performance indicators) in order to determine the progress of the initiative. In implementing and reviewing the measurement strategy, there are several factors that need to be taken into consideration. These include the amount of resources available, the amount and type of data and information that already exists and that which must be collected with new systems or sources, as well as the appropriate unit to collect the information. In developing the performance measurement strategy, every effort was made to use existing information sources, in order to reduce the costs and to minimize the collection burden within TPC. This performance measurement strategy is designed to be evergreen.

4.2 Evaluation

The purpose of this section is to outline the range of evaluation issues and questions to be used to guide the evaluation studies of the program. TBS guidelines for RMAFs note that evaluation issues should be identified. These guidelines, as well as the TBS evaluation policy and further TBS guidance, indicate that the evaluation issues should be categorized as follows:

The key issues that should be considered in evaluating the program are identified in the following pages according to these categories of issues. Given that this is a new program, a fifth category of issues focussed on implementation has been included as well.

Implementation

1) Is the design of the program appropriate given its objectives?

2) To what extent has the program been implemented as planned?

3) Have there been any unforeseen barriers / challenges to implementing the program as planned?

4) Are the communication and outreach activities reaching their intended audiences and having the desired impact?

Relevance

5) To what extent is the h2EA program fostering the development and early introduction into the marketplace in Canada of technologies that support the transition to the hydrogen economy?

6) To what extent has the national and internal environment changed since the program was created? Has the program evolved to meet the changing conditions? Are the program's objectives still relevant?

7) What role does the program play in the Technology and Innovation Initiative? Is there overlap or duplication?

Success / Progress

8) Are program objectives clearly understood by management and staff, co-deliverers stakeholders, clients and potential clients?

9) To what extent is the h2EA program achieving stated program objectives? Were there significant contributing factors and impediments to program success?

10) Who are the beneficiaries of the program? To what extent is the program reaching the intended target firms, including SMEs?

11) How have co-deliverers benefited from working with the h2EA program?

12) To what extent have funded projects achieved objectives identified in proposals and contribution agreements? How about technical and commercial success? What have been the environmental impacts of funded projects? What other impacts have occurred?

13) To what extent have the impacts of funded projects extended beyond the funded firm to the larger industrial community?

14) Were there any unintended outcomes?

Cost-Effectiveness

15) Are there alternative, more cost-effective ways of achieving the stated outcomes?

16) What changes could be made to improve the performance and likelihood of success for the h2EA program?

17) How effective are the governance structure and channels of communication for management and delivery?

18) Are the roles and responsibilities of the program and the co-deliverers clearly delineated and understood?

Results-Based Management

19) How effective is the on-going performance framework in providing a complete picture of the program's performance?

20) Has management made appropriate use of on-going performance measurement information?

21) How well is the on-going performance framework being used in planning and decision-making at the program and project level?

It is important to note that some of these issues may be excluded at the time of the evaluation and others may be added in order to better reflect the situation at that time.

The table below shows the proposed evaluation schedule for the h2EA program.

Table 2 - Proposed Evaluation Schedule
Evaluation Timing Focus
Six Month Review 6 months after start-up Issues related to implementation particularly outreach and communication
Formative Evaluation Q1 2005 Issues related to implementation, early success and results-based management
Summative Evaluation Q4 2007 Issues related to rationale, success, cost-effectiveness and results-based management

4.3 Integrated Ongoing Performance Measurements and Evaluation Strategy

It is critical to understand the links between the performance measurement and evaluation strategies, especially as it relates to the h2EA program. Ongoing performance measurement collects data on early outcomes and some intermediate outcomes at specific points in time and over a number of years. Evaluation allows for more in-depth analysis of the achievement of program outcomes. It includes the outcome data collected through the ongoing performance measurement strategy. These more complex issues are examined in an evaluation study in which more comprehensive and complex approaches can be used to better understand the success of an initiative, and the underlying factors influencing success. The evaluation also addresses questions concerning the relevance and cost-effectiveness of an initiative.

In the case of TPC and the h2EA program, the small number of demonstration projects - their formation, planning, implementation, monitoring and progress reporting - will form the basis for most of the on-going performance measurement of this program. Evaluation will be used to complement the project-oriented reporting on a periodic basis - at the interim and final stage of the program cycle covered by the requested funding.

The proposed integrated performance measurement and evaluation strategy identifies the performance indicators and the information which is required to effectively monitor the performance of the key results and sub-results at any particular time. The integrated performance measurement and evaluation strategy also outlines the indicators that will be collected or addressed through the periodic evaluation activities. This strategy also identifies how often the information needs to be gathered and by whom. The process and particulars of the strategy are provided in the following table.

Table 3 - Integrated Performance Measurement and Evaluation Strategy
Performance Area Indicator Data Source Responsibility Frequency
Immediate Outcomes        
Quality, seamless service to clients and co-deliverers is delivered Extent to which TPC consults with co-deliverers

Evidence of TPC coordinating with appropriate OGDs
Records of meetings, MOU, e-mails TPC Ongoing
Opinion of OGDs, co-delivers, stakeholders, clients Interviews, surveys TPC Periodic (Evaluation)
Increased awareness / acceptance of program by target clients, stakeholders and the public Level of awareness of program among target firms Record of enquiries

Survey of target firms
TPC Periodic
Level of acceptance of program among target firms Survey of target firms TPC Periodic
Correspondence with key stakeholders

Records of media commentary
TPC Ongoing
Appropriate uptake of the program Interest in the program by target firms

Interest in the program by key stakeholders
Record of enquiries

Correspondence with key stakeholders
TPC Ongoing
Number of quality applications

Range of interest (regional, public/private)
Internal program files

Correspondence with key stakeholders
TPC Ongoing
Partnerships / coalitions of companies working together are developed Number of partnerships / coalitions developed

Number of target firms participating in partnerships/coalitions
Contribution agreements

Program and project files
TPC Ongoing
Demonstration projects successfully implemented Number of proposals developed

Number of proposals accepted and funded
Contribution agreements

Program and project files
TPC Ongoing
Increased knowledge in demonstration participants Knowledge of target firms Periodic survey of participants TPC Periodic
Nature of projected project deliverables from funded demonstration projects Project progress reports

Internal TPC project files
TPC Periodic
Development of new firm capabilities Analysis of new capabilities developed in firm through project Application files, project files

Final project reports
TPC
Project participants
Ongoing / Annual
Opinion of participants, stakeholders and co-delivers TPC Periodic (Evaluation)
Codes, regulations, standards and guidelines are identified and developed Number of codes / standards that are identified

Evidence of Canadian standards that become adopted by international agencies
Internal program files

Industry press

Decision records of standards bodies meetings

Correspondence with stakeholders

Survey
TPC Periodic (Evaluation)
Strategic alliances are fostered and developed Emergence of new firms

Emergence of new joint ventures between firms
Project files

Business media reports

Correspondence with target firms
TPC Ongoing
Identification of R&D gaps and new research needs and elements of governance structure Number of new R&D projects related demonstrations (R&D) TPC applications TPC Ongoing
Survey of participants TPC Periodic
Intermediate Outcomes
Trades persons and companies build skill capacity Development of curriculum in universities and colleges

Development of qualified tradesmen (to service hydrogen applications)
Periodic literature search and/or survey TPC Periodic (Evaluation)
Companies, universities, and community colleges build technology capacity Development of curriculum in universities and colleges Periodic literature search and/or survey TPC Periodic (Evaluation)
Analysis of extent of project success (exceeded, met, met most, met some of target) Project files TPC Ongoing
Adoption of technology beyond funded firms (e.g., spin-off technologies) Number of unfunded firms and projects that become involved in a demonstration Business media reports, surveys, evaluation case studies TPC Ongoing and periodic (Evaluation)
Strengthened industrial capability in Canada and build community critical mass Number of firms that join a cluster

Growth or size and number of firms within a cluster
Business media reports, evaluation case studies TPC Ongoing and periodic (Evaluation)
Relative cost-effectiveness of different alternatives is improved Cost / kWH for different energy alternatives
Unit costs declining
Proponents through contribution agreements TPC Periodic (Evaluation)
An integrated supply of hydrogen technologies and hydrogen-compatible technologies is developed Breadth of participation (from supplier to end-user) in demonstration projects

Number of different types of participants in demonstration projects
Internal program files on projects

Business media reports (re. New and unfunded participants)
TPC Ongoing
Increased awareness and understanding, public confidence of hydrogen technology and hydrogen-compatible technologies Number and nature of articles written in the media

Number of questions that get raised with the Minister

Number and nature of related Qs & As in Parliament
Files on media reports

Qs & As

Internal files on enquiries
TPC Ongoing
Opinion of general public Surveys, focus groups TPC Periodic (Evaluation)
Development of new non-polluting sources of electricity is influenced Announcements of new projects which do not use fossil fuels Media reports TPC Periodic (Evaluation)
Ultimate Outcomes
Accelerated market acceptance of hydrogen technologies Number of hydrogen services and products that are launched

Sales of hydrogen products
Internal files

Media and business press

Surveys
TPC Periodic (Evaluation)
World-class talent attracted and retained Number of skilled labour in hydrogen cluster

Number of PhDs in related fields

Number of researchers
Surveys

Literature search
TPC Periodic (Evaluation)
Domestic and foreign investment in Canada attracted Growth of clusters in terms of number of firms and size of firms Media and business press

Surveys
TPC Periodic (Evaluation)
Contribution to the objectives of Canada's Climate Change Agenda Linkage of the h2EA program outcomes and objectives to Canada's Climate Change application of SMART, particularly replication potential agenda Analysis of program files, logic models TPC Periodic (Evaluation)
Opinion of key stakeholders Interviews TPC Periodic (Evaluation)
Potential long-term GHG reductions by demonstration projects1 Project files TPC Periodic (Evaluation)

1 Order of magnitude estimates would be used in this case (e.g., through the use of the SMART model). Detailed modelling would be beyond the scope of this project.

4.4 Methodologies

In accordance with best practices, the proposed approach for future evaluation studies involves the use of multiple lines of evidence and complementary research methods. The suggested methods make use of the ongoing performance measures as well as a number of additional methodologies. These methodological approaches are discussed in more detail below.

5.0 Reporting Strategy

The h2EA program will rely on the following reporting strategy:

The results of on-going performance measurement will be used to make the necessary adjustments to the initiative. In addition, the performance measurement strategy will be reviewed on an annual basis. Adjustments will be made as required to ensure that the performance information is appropriate and useful for on-going management requirements.

Date modified: