Evaluation of the Federal Government's Participation in the Toronto Waterfront Revitalization Initiative

| TOC | Previous | Next |

3.0 EVALUATION DESIGN

3.1 Purpose and Scope

The scope and design of the Evaluation of the Federal Government's Participation in the Toronto Waterfront Revitalization Initiative was based on preliminary work undertaken by the federal TWRI Secretariat, including the existing October 2006 Results-based Management Accountability Framework (RMAF). The evaluation matrix used to inform the evaluation design of this study is presented in Annex 1 of this report.

The evaluation assessed the relevance, success, design, delivery and cost-effectiveness/alternatives of the federal component of the TWRI since its commencement in 2000-2001. The evaluation was designed to determine whether the initiative is:

The scope of the evaluation was the approximately $410 million commitment of the federal government to the TWRI.

3.2 Evaluation Approach and Methodology

The Evaluation of the Federal Government's Participation in the Toronto Waterfront Revitalization Initiative was undertaken using multiple research activities. These activities included a document/literature review, research into comparison sites, a file review, key informant interviews, a survey of businesses, and focus groups with community stakeholders and residents. Each of these activities is described in the following sections.

3.2.1 Document and Literature Review

Members of the project team conducted a review of existing documentation from the three orders of government, the TWRC and other sources. The document and literature review was designed primarily to address the following research questions:

Numerous documents were also used to help assess the success of the TWRI, including the extent to which projects had resulted in expected outcomes.

The documents reviewed included:

In addition to government and TWRC documents, the consultant reviewed studies and research on the Toronto waterfront revitalization.

Extensive documentation on the TWRI was provided by the federal TWRI Secretariat at the commencement of the project. Further documentation was identified during key informant interviews with stakeholders from the three orders of government and the TWRC, as well as with experts in waterfront development. Annex 2 contains a list of documents that were consulted.

3.2.2 Research into Comparison Sites

The consultant undertook a review of other waterfront revitalization initiatives in other jurisdictions. Alternative initiatives were examined in order to provide points of comparison for the evaluation issues/indicators. Major evaluation questions addressed in the review included the following:

This review examined waterfront revitalization initiatives in Halifax and Winnipeg (at the Forks), as well as initiatives outside of Canada, including London, United Kingdom, and Sydney, Australia. Comparison sites were selected based on the availability of information and the comparability of the sites with the TWRI. This review of other initiatives was also informed by a report prepared in 2004 for the TWRC by Mercer Delta Organizational Consulting and entitled Review of Alternative Governance Structures and Delivery Models. Documents related to comparison sites were accessed through Internet searches and through telephone and email contact with stakeholders from the sites, as available.

3.2.3 File Review

In addition to a document and literature review, the consultant undertook a review of TWRI project files. The file review included a review of federal TWRI Secretariat and TWRC project files for 28 federally funded projects, representing a total of 32 contribution agreements. File reviews included reviews of the contribution agreements and project activity reporting, as well as communication material and correspondence related to project activities, as identified. File reviews were undertaken using a structured guide that listed the relevant evaluation data to be captured. The major evaluation issues addressed in the file review included the extent to which the TWRI projects:

3.2.4 Key Informant Interviews

In order to gather in-depth information on many of the central evaluation questions and to supplement information collected through the document and file review, the research team conducted key informant interviews with representatives from the following groups:

The interview sample was developed in consultation with Environment Canada in order to include a range of stakeholders knowledgeable about the TWRI and Toronto waterfront revitalization generally. While interviews were undertaken with all targeted stakeholder groups, some potential respondents were not available, including some members of the TWRI IGSC. However, completed interviews provided a broad range of perspectives on the TWRI.

Interviews were undertaken using semi-structured key informant interview guides, customized for each respondent group (including three separate guides for federal Toronto Waterfront Revitalization Initiative Secretariat members, other Initiative stakeholders outside the federal government, and academic experts). The guides were customized to ensure that respondents were asked to provide input on evaluation issues that matched their areas of knowledge. Interviews were undertaken in person or by telephone, depending on the availability and preference of the respondent.

Additional contact was also made with a representative of the City of Toronto's Planning Division in order to obtain data related to employment and land-use in the waterfront area, and with the federal TWRI Secretariat's environmental assessment consultant for information related to the environmental assessment process.

3.2.5 Survey of Businesses

A survey was conducted of businesses operating in the Toronto waterfront area. The survey addressed issues related to the impact of the TWRI, including:

The survey questionnaire was pre-tested to ensure that questions were clear and generated the intended information, and that the order and flow of questions was appropriate.

The sample was developed using the infoCanada database of businesses. While the sample included businesses from all types of industries, it included an over-sampling of businesses more likely to be affected by waterfront revitalization activities, including retail and service industry businesses. Geographic parameters were established prior to sampling to ensure that the survey targeted those businesses that operated in the waterfront area. The following forward sortation areas (FSAs), derived from the first three characters of postal codes, served as the boundaries for the sample: M4M, M5A, M5E, M5J, M5V, M6K, M5W. (A map showing the location of these FSAs is provided in Annex 4). The valid response rate for the survey was 25%, which is comparable to response rates in other business surveys undertaken by the research consultant.

Analysis included the examination of statistical significance of survey responses by industry groups and for different areas of the waterfront, using chi-squares.

3.2.6 Focus Groups

Focus groups were moderated with a sample of residents and representatives of community organizations (citizens' groups, housing organizations, environmental stakeholders). Three focus group sessions were held, including two in which the focus groups comprised residents of the waterfront area (with seven participants in each group) and one in which the focus group participants were members of community and neighborhood associations in the waterfront area (with 13 participants representing 12 different organizations). Focus groups were hosted at dedicated focus group facilities, and discussions were audio- and video-taped. Analysis of focus group feedback consisted of summaries of the responses obtained in each focus group session, organized by evaluation issue.

The purpose of these focus groups was primarily to provide in-depth qualitative information on selected evaluation issues related to the level and impact of community involvement and interest in the TWRI, and any unintended impacts of the TWRI on residents and the community.

3.2.7 Research Challenges and Limitations

Challenges and limitations for the evaluation project were identified as follows.

Isolating and attributing the results and success of a tri-government funded program/initiative: The TWRI includes the participation of three orders of government and the TWRC. This evaluation, however, focused solely on the participation of the federal government in the TWRI. It was often difficult to isolate the success and effectiveness of the federal component of the TWRI from that of the other participating governments and the TWRC.

Further, the Toronto waterfront is a large area, which has involved a range of stakeholders, activities and types of development. As a result, it was difficult to determine to what extent businesses, residents, and community groups consulted for this evaluation were able to isolate federally funded TWRI activities from other TWRI activities, as well as from non-TWRI activities in the waterfront area.

The direct impact of much of the TWRI work completed to date is difficult to assess: A significant portion of the TWRI projects completed or commenced are related to planning, design or preparation of land for future development. Given the nature of these activities, little of the work completed to date has had a measurable impact on the waterfront area, including on its businesses or residents. As a result, many of the expected outcomes of the TWRI outlined in the federal government's performance strategy for the TWRI-including, for example, increased economic development and increased usage of the waterfront area-cannot be expected to evolve directly from much of the work completed to date. However, some project activities, including the completion of improvements to John Quay and York Quay and the construction of the Western Beaches Watercourse Facility, for example, have resulted in improvements to waterfront usage and accessibility.

| TOC | Previous | Next |