Archived—Six-Month Progress Report

Archived

Final Report

Submitted to:

Industry Canada / Technology Partnerships Canada

Submitted by:

Performance Management Network Inc.

November 23, 2004




Table of Contents




Executive Summary

This report summarizes the progress of the hydrogen Early Adopters (h2EA) program – launched in the Fall of 2003. The report focusses on the first six months of progress, with some reference to progress through August of 2004.

Implementation

The h2EA is recognized as a program which is "pushing the envelope" in terms of attempting to proactively foster hydrogen communities in Canada. For this reason it embodies several features which have caused some barriers to a smooth start-up. These include:

Success / Progress

Given the implementation issues, h2EA has made reasonable progress. Understanding of objectives has evolved over time, appropriate audiences appear to have been eventually reached and co-deliverers are emerging in their program support roles.

Cost-effectiveness

The governance and role definition of h2EA continues to evolve, as IC regional offices, regional development agencies and others begin to feel more involved in the process. The program's designation of specific officers for projects has helped the process. Despite what has generally been regarded as strong, professional and responsive support from h2EA staff – there is a concern that the program is under resourced to handle the load for this small, but highly complex and innovative program.

Results-based Management

While the RMAF, and the early progress review (i.e. this study) were seen as positive features, there is concern re: the clarification of roles and dedication of resources to monitor further progress in the program. The most promising approach would seem to be one which engaged project stakeholders in case by case tracking of key desired outcomes such as technical milestones, collaborative achievements, target audiences reached, codes and standards impacts, sustained behavioural adaptations (if applicable) and where possible, quantitative benefits in terms of energy, wastage / pollution and / or cost savings.

Recommendations

In order to enhance the progress of h2EA over the coming months, the following recommendations are offered. TPC management should:

  1. Clarify h2EA's niche vis a vis other alternative energy and hydrogen initiatives such as SDTC and CTFCA.
  2. With respect to item # 1, review the resource loading, roles, mandate and the implications of these on all delivery players – including h2EA, TPC, SDTC, CTFCA, Industry Canada regional offices and other partners.
  3. Pursue ways and means to improve channels of communication relating to program developments, application requirements, eligible activities and costs, and useful / important learning from the first year of operations.
  4. Enhance the performance planning and measurement framework with more explicit recognition of progress among co-delivery and intermediary communities, and the build-up of a network relating to h2EA adaption. (Examples are provided in this report – see Figure 1, Appendix D and Appendix E.) Elements of the measurement framework will need to be applied on a case by case basis – given the diversity of the projects at hand.
  5. In order to achieve the above, TPC should pursue ways and means to increase the resources devoted to h2EA program management.

1.0 Introduction

1.1 Objective and Scope

The objective of this report is to review the progress of the hydrogen Early Adopters Program – launched in the fall of 2003. This report addresses two goals:

  1. It provides a progress report focussed on the first six months of the initiative – addressing the intended results and issues established in the 2003 Results-based Management and Accountability Framework (RMAF); and,
  2. The report reviews the results logic and the appropriateness of the previously established framework and measurement strategy. It makes suggestions as to future adjustments.

The scope of this report involves an analysis of progress over the first six months of the h2EA initiative. For this reason, the focus is on early activities and outputs, as well as the engagement and support of key collaborators – and the reach and early influence of the h2EA target community.

1.2 The h2EA Initiative

A full program profile for h2EA is contained in the Results-based Management and Accountability Framework – filed in the summer of 2003. This section summarizes the profile information.

The Program is aimed at addressing the urgent need to accelerate the market adoption of H2 technologies and other H2 compatible technologies that facilitate the transition to a hydrogen economy and attracting world-class talent and investment to Canada. Through Technology Partnerships Canada (TPC), support will be given to the establishment of integrated hydrogen complexes such as "H2 Villages", "H2 Highways" and other similar partnerships.

This support will be targeted to multiple public- and private-sector partners to demonstrate theses technologies and showcase Canadian capabilities. These partnerships will involve integrating H2 compatible technologies and hydrogen production, storage and distribution technologies with fuel cell and related portable, stationary and mobile applications in a microcosm of the hydrogen economy. Results of this initiative will include real-world experience and expertise; early market adoption of H2 technologies and infrastructure needed to support their wide-spread use; and increased consumer and investor awareness. A total of $60M has been allocated from Budget 2003 for this initiative, of which $10M has been made available by TPC to the Canadian Transportation Fuel Cell Alliance to allow it to strengthen or extend its ability to meet the objectives of the Early Adopters Program.

In terms of the intended delivery, h2EA was intended to be implemented by Industry Canada through Technology Partnerships Canada (TPC) in conjunction with the regional development agencies and other federal departments and agencies and their respective programs. Memoranda of understanding may be developed between TPC and federal departments and agencies to support the implementation of this initiative.

During the period FY2003-04 to FY2007-08, a total of $10M was made available by TPC to the Canadian Transportation Fuel Cell Alliance (CTFCA) to allow it to strengthen or extend its ability to meet the objectives of the Early Adopters Program.

Regional development agencies were invited to participate in the building of partnerships and coalitions, considering and monitoring the projects, and communicating the successes of regional initiatives.

Experts from appropriate Industry Canada sectors have been involved in the due diligence process.

Other federal departments and their respective programs were also invited to participate, where their program objectives complement those of the Early Adopters Program.

As noted above, the reach of the h2EA initiative follows a progression. Over the early stages, the program reaches Industry Canada groups (i.e., sector branch and regional office staff), followed by a broader group of agencies within the federal government and other intermediaries. As these liaisons and networks are built, the initiative begins to reach key members of its primary target community – the groups representing important actors in the fledgling hydrogen community. From this group, the intent is to reach broader communities of interest – including the Canadian public. Figure 1 represents a reach and results logic for the initiative. It is derived from the 2003 RMAF – embellished to include a more complete picture of h2EA reach and results-based on experience to date.

Figure 1: The Reach and Results of h2EA

Figure 1: The Reach and Results of h2EA

2.0 Methodology

This progress report included the following methodologies:

Note that, after consulting TPC management, a survey of participants / applicants was not undertaken at this time.

Each methodology is described in the sections which follow.

2.1 Document, Database, File Reviews

In the context of this six month review, it was important to review information from existing sources such as program and project documents and files as well as the information in the program database. These included:

This approach provided valuable information on program implementation as well as program reach.

2.2 In-depth Interviews

A selection of in-depth telephone interviews were completed to collect information on program design (objectives, strengths, weaknesses, improvements), barriers or challenges, communication and outreach activities and early program success.

2.3 Analysis and Reporting

The above sources were analyzed with a view to addressing key questions outlined in the study planning report, as well as being used to review the results story and measurement strategy for h2EA.

3.0 Findings

The findings of this report follow the issues established by the 2003 RMAF and are addressed in the following sections.

3.1 Implementation

The implementation of the h2EA initiative was conducted by TPC over the first six months as follows:

Pre-October 2003 √ Preparatory consultation, design development
October 2003 √ Announcement of program
√ Simultaneous release of program Terms and Conditions, explanatory material, "how to Apply" documentation – all released on the web expenditures in year 1.
October 2003 to March 2004 √ Liaison with partners and intermediaries
  • Hydrogen Fuel Cell Committee (meetings)
  • Canadian Transportation Fuel Cell Alliance (meetings)
  • Sustainable Development Transportation Committee (meetings)
  • Canadian Hydrogen Association
November 2003 to February 2004 √ Information sessions held in Montreal, Toronto, Saskatoon, Calgary, Edmonton, Vancouver and Charlottetown
√ 3 phased approach I) IC regional offices; ii) Government Partners; iii) Primary Target Community and public
November 2003 to February 2004 √ Application intake
√ Receipt of applications
√ Refinement

3.1.1 Is the design of the program appropriate given its objectives?

Early feedback suggests that the design of the program is generally appropriate given its objectives, however there are items which required refinement in the first 6 month period.

  1. Optimistic Community 'Readiness'

    The program was designed with significant funding to spend in a short time frame. Given the need to foster awareness, understanding and collaboration among a disparate set of players – there is some evidence that it would have been useful to have had a design which would have rolled out over a slightly longer 'sunrise' time frame. In other words – given the wide scope and cause-effect complexity, as well as the transformative effect desired (i.e., the need to foster a collaborative hydrogen community) it appears to have been optimistic to have laid out a design which anticipated the significant flow of contribution expenditures in year 1.

  2. Liabilities for Consortia

    Early applicants appear to have struggled to address questions of liability posed by TPC during the application process. The design of agreements requiring joint and several liability among consortium partners appears to form a barrier to progress. Liability issues have been found to surround other TPC programming (e.g., This has been an issue for the larger, mainstream TPC program as noted in the 2003 evaluation study.) and, if anything, it looms even larger for many members of the fledgling Canadian hydrogen sector.

  3. Clarity of Roles and Responsibilities

    There is some evidence that the design of the program may not have anticipated the full extent of the support roles and responsibilities to be played by all co-delivery partners –including other government departments, various 'alliances' (e.g. SDTC, CTFCA) and IC regional officers. (See also section 3.3.2)

3.1.2 To what extent has the program been implemented as planned?

The program released information on its website simultaneous to its October announcement. This showed a strong harmony in terms of specific information supporting the general announcement.

The information sessions were generally conducted in the November-December period – essentially as planned – with the exception of one session extending into February 2004.

Applications for the first wave were slower than expected with only two applications submitted for the November 28th initial deadline. The process picked up steam with the second and third waves however, resulting in more than 40 applications by February 16, 2004.

In general, submissions have been refined and consolidated – in discussions among h2EA program officers, regional officers and hydrogen sector proponents.

3.1.3 Have there been any unforeseen barriers / challenges to implementing the program as planned?

One area of unforseen barriers has been that of internal application processing and approvals. Questions and uncertainties emerging as parts of the standard TPC application process – such as issues relating to liabilities and (see section 3.1.1) revenues and assets arising from the demonstration projects, questions regarding the treatment of capital equipment and potential repayability appear to have slowed down the approval process.

3.1.4 Are the communication and outreach activities reaching their intended audiences and having the desired impact?

The strong eventual response of various groups, some of whom were not initially 'known' to be interested in these areas – but who eventually assembled strong consortium proposals (names withheld due to pending contribution negotiations) suggests that program communications were successful in widening the 'net' of potential actors in Canada's hydrogen clusters.

In terms of communications with partners and co-deliverers, there appears to have been some gaps in clarity. It was noted that assistance in the delivery of h2EA was not included in the HQ-Regional TPC Memorandum of Understanding.

3.2 Success / Progress

3.2.1 Are the program objectives clearly understood by management and staff, co-deliverers, stakeholders, clients and potential clients?

Program objectives appear to be clearly understood by most management and staff – as well as among government co-deliverers. A detailed analysis of the understanding of objectives among clients and potential clients was beyond the scope of this review, however it appears that the need to demonstrate the workings of hydrogen initiatives has been reasonably well understood by all parties.

3.2.2 Who are the early beneficiaries of the program? To what extent is the program reaching the intended target firms, including SMEs?

An analysis of the first three approved projects shows the following:

Project #1: Fuel Cell Technology

Lead recipients and partners
Fuel Cells Technology: Technology provider
Ontario Power Generation: System integrator
University of Toronto: End-user
Participants
Enbridge Gas Distribution: Energy supplier
Air Liquide Canada: Energy supplier

Project #2: Hydrogenics

Lead recipients and partners
Hydrogenics: Technology provider
John Deere e-Power Technologies: System integrator
Greenlight Power Technologies: Technology provider and system integrator
Marconi Corporation: System packaging and installation
Participants
City of Toronto: End-user
Toronto Renewable Energy Co-operative: Technology partner
Canadian Fuel Cell Alliance: Refuelling station support
Bell Canada: Technology adopter and system integrator
Purolator Courier: End-user

Project #3: Ballard

Lead recipients and partners
Ballard: Technology provider
MGE: Technology integrator and distributor
Bell Canada: End user, technology adopter
Marconi: System packaging and installation
Praxair: Hydrogen storage and distribution
IBM: End user, technology adopter and system integrator
UTM: End-user, technology adopter and system integrator

Other projects display a similarly wide and in some cases even wider diversity in partners and participants.

3.2.3 How have co-deliverers benefited from working with the program?

It is early in the process to pronounce on co-deliverer benefits. The role of co-delivery partners and communications with them have evolved extensively over time.

3.3 Cost-effectiveness

3.3.1 How effective are the governance structure and channels of communication for management and delivery?

Collaboration among members of the Hydrogen and Fuel Cell Committee appears to be progressing well.

Some early concerns have emerged in both governance and communications.

In terms of governance of the contributions, the approval process appears to have emerged as a bit more lengthy and difficult than initially anticipated.

In terms of communications – while the program made a strong effort to release information at the time of the program announcement, there were some aspects of program which appear to have remained unclear including:

3.3.2 Are the roles and responsibilities of the program and the co-deliverers clearly delineated and understood?

The roles and responsibilities of all delivery partners and collaborators appears to have shown some ambiguity during the early months of the program roll-out. Regional IC officers in some cases appear to have been placed in roles which they may not have anticipated as interlocutors between project proponents and TPC management.

3.3.3 What changes could be made to improve the performance and likelihood of success for the h2EA program?

The resourcing for the delivery of this initiative – much like the resourcing for the delivery of TPC in general – appears to be somewhat light given the task at hand – to foster networks and to build hydrogen communities. The scope and cause-effect complexity of this effort combined with the diverse nature of applicants – suggests that a good deal of patient hand-holding is likely in order to get projects off of the ground. The three Ottawa-based TPC staff dedicated to delivery, while apparently capable and well intentioned – may not be sufficient to acquire, disseminate and manage all of the information and knowledge required to address the needs of the program at this time.

3.4 Results-based Management

3.4.1 Based on the way in which the program was implemented, are there any changes required to the performance measurement and evaluation strategy identified in the h2EA program RMAF?

Analysis of h2EA's first 6 months of progress suggests that a key element in the successful implementation of a fledgling initiative such as this is the establishment of a supportive network – not just in the target communities – but also among policy-rule (e.g. codes and standards) makers, delivery partners and key intermediaries. The results logic and measurement scheme for h2EA needs to be adjusted to accommodate important achievement in awareness, understanding and support among the co-delivery community. This has been done in Figure 1. Annex D suggests some key additions to the performance measurement strategy. Annex E goes further, and suggests a social network analysis approach for future evaluation work.

4.0 Conclusions and Recommendations

This report summarizes the progress of the hydrogen Early Adopters Program (h2EA) – launched in the Fall of 2003. The report focusses on the first six months of progress, with some reference to progress through August of 2004.

4.1 Conclusions

Implementation

The h2EA is recognized as a program which is "pushing the envelope" in terms of attempting to proactively foster hydrogen communities in Canada. For this reason it embodies several features which have caused some barriers to a smooth start-up. These include:

Success / Progress

Given the implementation issues, h2EA has made reasonable progress. Understanding of objectives has evolved over time, appropriate audiences appear to have been eventually reached and co-deliverers are emerging in their program support roles.

Cost-effectiveness

The governance and role definition of h2EA continues to evolve, as IC regional offices, regional development agencies and others begin to feel more involved in the process. The program's designation of specific officers for projects has helped the process. Despite what has generally been regarded as strong, professional and responsive support from h2EA staff – there is a concern that the program is under resourced to handle the load for this small, but highly complex and innovative program.

Results-based Management

While the RMAF, and the early progress review (i.e. this study) were seen as positive features, there is concern re: the clarification of roles and dedication of resources to monitor further progress in the program. The most promising approach would seem to be one which engaged project stakeholders in case by case tracking of key desired outcomes such as technical milestones, collaborative achievements, target audiences reached, codes and standards impacts, sustained behavioural adaptions (if applicable) and where possible, quantitative benefits in terms of energy, wastage / pollution and / or cost savings.

4.2 Recommendations

In order to enhance the progress of h2EA over the coming months, the following recommendations are offered. TPC management should:

  1. Clarify h2EA's niche vis a vis other alternative energy and hydrogen initiatives such as SDTC and CTFCA.
  2. With respect to item # 1, review the resource loading, roles, mandate and the implications of these on all delivery players – including h2EA, TPC, SDTC, CTFCA, Industry Canada regional offices and other partners.
  3. Pursue ways and means to improve channels of communication relating to program developments, application requirements, eligible activities and costs, and useful / important learning from the first year of operations.
  4. Enhance the performance planning and measurement framework with more explicit recognition of progress among co-delivery and intermediary communities, and the build-up of a network relating to h2EA adaption. (Examples are provided in this report – see Figure 1, Appendix D and Appendix E.) Elements of the measurement framework will need to be applied on a case by case basis – given the diversity of the projects at hand.
  5. In order to achieve the above, TPC should pursue ways and means to increase the resources devoted to h2EA program management.

Annex A – List of Persons Consulted for this Report

The following people were consulted in the process of assembling this report. In some cases they deferred to others in submitting answers to our questions.

List of persons contacted for this report1:

Mark Romoff, Executive Director, Industry Canada, Ontario Region
Gerry Cooper, Special Consultant, Industry Canada, Ontario Region
Derek Yue, Innovation Regional Agent, Industry Canada, Ontario Region

Robert Sirois, Investment Director, Industry Canada, Quebec Region
Claude Morasse, Regional Innovation Agent, Industry Canada, Quebec Region
François-Nicolas Pelletier, Regional Innovation Agent, Industry Canada, Quebec Region

Genine McCurdy, Industry Director, Industry Canada, Vancouver
Annie Desgagné, Senior Commerce Officer, Industry Canada, Vancouver
Gordon Giles, Regional Innovation Agent, Industry Canada, Vancouver

Glenn Fields, Executive Director, Industry Canada, Prairie and Northern Region
Dee Pannu, Principal Consultant, Innovations, Industry Canada, Alberta

Claire Lepage, Executive Director, Industry Canada, Atlantic Region
Patricia Hearn, Provincial Director, Industry Canada, Atlantic Region
Gailene Murphy, Regional Innovation Agent, Industry Canada, Atlantic Region

Kathryn Bruce, Director General, TPC, Industry Canada, Ottawa
Simon Brault, Director, Economic and Business Case Analysis Directorate, Industry Canada, Ottawa
Jacques Cloutier, Senior Investment Manager, Industry Canada, Ottawa
Sylvain Caron, Investment Officer, Industry Canada, Ottawa
Doris Dupuis, Assistant, Industry Canada, Ottawa

Rick Whittaker, Vice President, Sustainable Development Technology Canada, Ottawa
Richard Fry, Manager, Canadian Transportation Fuel Cell Alliance, Natural Resources Canada, Ottawa

Annex B – List of Documents Consulted for this Report

List of documents consulted for this report:

  1. Results Based Management and Accountability Framework (RMAF), Industry Canada – Technology Partnerships Canada Early Adopters Program
  2. Hydrogen Road Map for Canada, Vision and Mission, Meeting the Greenhouse Gas Challenges – Kyoto and beyond: Canadian Leadership in the Hydrogen Age, Natural Resources Canada
  3. Project Application files
  4. Canadian Fuel Cell Commercialization Roadmap
  5. NRC Hydrogen and Fuel Cell Program Presentation
  6. NSERC Research Partnerships Programs Presentation
  7. Website

Annex C – Original h2EA Logic Model from 2003 RMAF

Figure 3 - Early Adopters Program Logic Model Objective: To work in partnership with hydrogen industry stakeholders to foster the demonstration and early introduction into the marketplace in Canada of technologies, such as fuel cells and those used to produce, store and distribute hydrogen, that support the transition to the hydrogen economy.

Figure 3 - Early Adopters Program Logic Model Objective

Annex D – Suggested Performance Measurement Framework Additions

Figure D-1: Performance Framework Additions
Performance Area Indicators Data Source / Methods Responsibility for Collection Timing / frequency of measurement
Coordination with OGDs type / nature of relationships with OGDs* and alliances (e.g.SDTC, CTFCA) Content analysis of meeting minutes / correspondence

Limited survey
TPC Ongoing
Awareness and Understanding Level of communication with potential partners*

Quality and timeliness of correspondence with potential and actual partners*
Content analysis of correspondence with (potential) partners

Limited survey
TPC Ongoing
Collaborative support of OGDs, as required # of OGDs providing information and support*
Ways in which OGDs have provided support
Contribution agreements, TPC files / content analysis

Qualitative assessment of support provided

Limited survey
TPC /supporting Department / Agency Ongoing
Resources leveraged from partners $s leveraged towards funded project (actual vs. proposed)*
Level of compliance
Content analysis of contribution agreements

Limited survey
TPC Periodic, at time of signing (proposed) and at time of evaluation (actual)

* See Annex E for suggested tracking of the social 'network' as a means of tracking h2EA progress.

As noted in Section 3.4, h2EA's early progress suggests that it will be important to understand the level and extent of collaboration achieved with various partners and stakeholder communities – as a key to understanding program progress.

The performance areas, indicators and sources / methods listed above represent suggested measures which could be integrated into the existing RMAF performance measurement strategy.

Annex E – Implications of Our Findings for h2EA Progress Reporting and Evaluation

During the course of the data collection and analysis involved in assembling this progress report, it has become clear that the unique nature of the h2EA initiative requires a reconsideration of evaluation methods. Just as conventional project approval 'mindsets' and criteria are not appropriate for this program (see section 4.1 – Conclusions on Implementation), a conventional evaluation approach is likely not warranted.

The fact that h2EA is essentially in place to accelerate the adoption of hydrogen technologies by fostering networks and infrastructure, suggests that the evaluation approach applied to understanding its success should adapt to this program theory. The approach should not focus on establishing 1st and 2nd order impacts on firms per se, but rather should start by establishing the degree to which a successful network of collaborations is taking place. In order to do this, a 'social interaction' or 'network analysis' view of h2EA is required.

J. T. Landry, in the November 2004 Harvard Business Review stated, "Innovation is a social process; the more successful the innovation, the more social the process."

Recent work commissioned for the U.S. Department of Energy (DOE) suggests that it is important to understand the 'social' innovation system within which a technology is to be deployed. Knowledge of the system is necessary for gauging the commercialization prospects in a given area (e.g. for hydrogen). Figure E-1 depicts an innovation system based on social interaction. The DOE has concluded that "Mapping social relationships in an innovation system can be useful in facilitating technology development and in assessing impacts."

In our view, this kind of mapping may be vital to understanding h2EA's progress. While a detailed analysis is beyond the scope of this study, it would appear that one possible approach to mapping h2EA's social networking and progress might go as follows:

  1. Categorize 2004 applicants for assistance by key (functional and geographic) communities (see Figure 1 for broad categories and section 3.2.2 for market players).
  2. Proceed to survey groups to establish their progress re: hydrogen initiatives, as well as their social networks – in addition to perceptions re: program assistance. Use a limited 'snowball' survey approach2 to identify the extent of social networks being developed.
  3. Map out the social network of h2EA, perhaps attempting to show a bit of a time series (i.e. the size and diversity should build over time), as well as various linkages (type and frequency) by groups and communities.

Figure E-1: A Social Interaction View of Innovation

Figure E-1: A Social Interaction View of Innovation

Source: Adopted by Przybylinski et al., Temporary Organizations for Collaborative R&D: Analyzing Deployment Prospects, 2000, from Havelock and Havelock.

Derived from U.S. DOE and U.S. National Institute for Standards and Technology (NIST), A Toolkit for Evaluating Public R&D Investment Models, Methods and Findings from ATP's First Decade Ruegg and Feller, July 2003, p 132

The use of a social networking approach would nicely address certain key questions relating to the relative progress of h2EA, the patterns which emerge in different applications, key 'nodes', and the co-relation of different types of networks with various measures of success.

We estimate that a network analysis, built upon a planned formative evaluation survey methodology, could replace certain conventional monitoring approaches relating to benefits and impacts, and would therefore add little, if any cost to anticipated data collection efforts. (i.e. Certain questions would be added to an already anticipated survey and might replace certain 'conventional' questions – this would likely not add to the survey burden.)

Annex F – Review of Approved Project Reporting Requirements vs. RMAF Requirements

As part of the work for this study, the first three successful projects were reviewed for their potential to serve as a data source to address the information needs outlined in the 2003 Results-based Management and Accountability Framework.

The findings are summarized by key performance area and indicator below (note: only those areas listing internal project files as a potential source are included):

Performance Area Indicator Finding
Appropriate uptake of the program Number of quality applications

Range of interest (regional, public / private)
  • Evidence in the file is ample. The use of independent review (i.e. the Deloitte review) provides a ready assessment of quality.
  • The various stakeholders are clearly identified.
Partnerships / coalitions of companies working together are developed Number of partnerships / coalitions developed

Number of target firms participating in partnerships / coalitions
  • This is evident in the application and review documents on file.
Demonstration projects successfully implemented Number of proposals developed

Number of proposals accepted and funded
  • Available from the project management system.
Increased knowledge in demonstration participants Knowledge of target firms

Nature of projected project deliverables from funded demonstration projects
  • The project reporting requirements suggest that this be covered, however it is open-ended, and will likely need to be complemented by another line of evidence (e.g. a survey).
Development of new firm capabilities Analysis of new capabilities developed in firm through project
  • The project reporting requirements suggest that this be covered, however it is open-ended, and will likely need to be complemented by another line of evidence (e.g. a survey).
Codes, regulations, standards and guidelines are identified and developed Number of codes / standards that are identified

Evidence of Canadian standards that become adopted by international agencies
  • The project reporting requirements suggest that this be covered, however it is open-ended, and will likely need to be complemented by another line of evidence (e.g. a survey).
Strategic alliances are fostered and built Emergence of new firms

Emergence of new joint ventures between firms
  • The emergence of new firms was not explicitly suggested as an outcome, however the emergence of new ventures (alliances etc.) was noted. This will likely need to be complemented by another line of evidence.
Identification of R&D gaps and new research needs andelements of governance structure Number of new R&D projects related demonstrations
  • Elements of this will likely be picked up in reporting, however other lines of enquiry will likely also be necessary.
Trades persons and companies build skill capacity Development of curriculum in universities and colleges

Development of qualified tradesmen (to service H2 applications)
  • This is indirectly addressed in the reporting requirements as part of a menu of potential outcomes. It would almost certainly need a more direct line of enquiry.
Companies, universities, and community colleges build technology capacity Development of curriculum in universities and colleges

Analysis of extent of project success (exceeded, met, met most, met some of target)
  • Not explicitly addressed.
  • Project success will be at least partially assessed by project reporting. It will definitely require complementary methods.
Adoption of technology beyond funded firms (first purchase,spin-off technologies) Number of unfunded firms and projects that become involved in a demonstration
  • No reporting is required of unfunded applicants. Follow-up will require a separate survey.
Strengthened industrial capability in Canada and build community critical mass Number of firms that join a cluster

Growth or size and number of firms within a cluster
  • The project files will serve as a base for follow-up (See Annex E for description of proposed social networking approach).
Relative cost-effectiveness of different alternatives is improved Cost / KWH for different energy alternatives

Unit costs declining
  • Project reporting requirements ask for this type of reporting. This may need other lines of enquiry for verification.
An integrated supply of H2 technologies and H2 compatible technologies is developed Breadth of participation (from supplier to end-user) in demonstration projects

Number of different types of participants in demonstration projects
  • The project files clearly show the participants and partners. These can be categorized (see section 3.2.2) and analyzed for their relative diversity.
  • The 'outreach' components of project reporting requirements are left a bit vague in terms of specific target groups and expected numbers. This will likely require some complementary follow-up.

In summary, TPC h2EA project files appear to establish a reasonable base for project tracking – as anticipated by the 2003 RMAF. Some of the expected results are stated in terms which might be a little bit general. (e.g. A couple of descriptions of expected project results in form h2EA-1 make statements like 'increased knowledge' and 'expanded awareness' without operationally defining these terms to an extent where specific expectations or targets could be set.

Management may wish to review the need to provide specific targets as part of future approvals. After all, if a project is essentially for demonstration to increase awareness, the question of "how much awareness are we buying?" needs to be addressed.) Notwithstanding this potential gap, the files appear to set out reporting requirements which, if reasonably followed, would at minimum support and complement more detailed lines of enquiry in terms of tracking h2EA results.


1 While all of the above were contacted, not everyone on this list participated in an interview. In order to preserve sufficient anonymity, only the broader contact list is shown here.

2 The snowball survey approach would ask respondents for others they have had meaningful contact with in the development and implementation of their hydrogen project. This could apply to groups who did not receive assistance as well as those who did. The results of the first round of this questioning (i.e. the contacts) can also be contacted for their contacts, as well as for progress and perceptions, creating a 'snowball' effect. (See Figure E-1 for a graphic illustration.) For our purposes, we would likely limit the follow-up surveying so as to stay within reasonable resourcing limits.

Date modified: