Placemat - Evaluation of the Urban Programming for Indigenous Peoples program
Overview
- The program supports activities that allow Indigenous organizations to serve clients, and to deliver effective culturally appropriate programs/services to Indigenous Peoples in urban spaces, urgent infrastructure and housing supports to Indigenous persons residing in urban, rural, and northern areas.
- Five program streams were evaluated: Organizational Capacity; Programs and Services; Coalitions; Research and Innovation; and Infrastructure (added in December 2020).
Temporal
- The evaluation examined 2017-18 to 2021-22.
Financial
- Annual expenditures averaged $199.6 million (Including Covid-19 funding)
Evaluation Domains
- Relevance, effectiveness, and efficiency, as seen through the lens of the Awakening the Journey framework: a hybrid evaluation framework grounded in Indigenous perspectives.
Evaluation Framework
Awakening the Journey: hybrid approach that both adheres to the Treasury Board of Canada Secretariat Policy on Results and the exploratory research spearheaded by Johnston Research Inc. in partnership with ISC, which foregrounds Indigenous worldviews and community wellness. The hybrid approach uses five pillars across four periods of time (past, present, transition, future):
- Spirit: qualities which form the definitive character of a person, Nation, or group
- Relationships: interpersonal and spiritual connections that run through the program and evaluation
- Process: ways in which the program and evaluation conduct programming and carry out methodologies
- Governance: how processes align with Government of Canada legislative frameworks and priority areas
- Outcomes: responses to the needs and priorities of Indigenous Peoples and the Government of Canada
Methodology
- Data collection, data analysis, and engagement used a "methodological toolkit" approach. Inspired by the principles of the Awakening the Journey framework, the toolkit included both Indigenous and Western tools that could be applied flexibly from context to context. Tools evolved as the evaluation progressed to increasingly reflect guidance and input from partner organizations and the 12-member Technical Advisory Committee, comprised of diverse Indigenous partners from across Canada. Tools included (among others): interviews, sharing circles, artistic (symbol-based) reflection, communal meals, ceremonies, and community tours.
- The final evidence base includes data gathered from: 2 surveys, 55 Key informant interviews, 7 sharing circles, 2 focus groups, 4 community meals, 1 artistic (symbol-based) reflection activity, and 6 community tours. Contributors included ISC program staff; leadership and staff of Indigenous organizations who receive UPIP funding; users of UPIP-funded programs; and - to highlight gaps in service – organizations and Indigenous Peoples in urban spaces who did not receive UPIP funding.
Data Analysis
- Indigenous methodologies extend from how data are collected to how data are interpreted and presented. Indigenous approaches to data analysis often focus on narrative salience and collaborative sense-making to determine the strength and importance of findings, rather than on codification and quantification.
- The Awakening the Journey evaluation framework outlines consensus decision-making as a way of establishing key findings. In alignment with this approach, the evaluation team and contractors held three 3-hour sense-making sessions to identify prevalent themes and reach a consensus on key findings.
- Following this initial consensus-based process, another sense-making session was held with the Technical Advisory Committee, who provided feedback to triangulate findings and reach consensus between ISC Evaluation, the contractors, and Indigenous partners.
- The key findings of this evaluation are those which were determined by consensus to be most salient and are those which have led to recommendations and next steps
Key Findings
- Spirits : UPIP's flexibility in delivery allows Indigenous service providers to integrate diverse Indigenous values and ways of knowing into their work, which is a significant strength. However, the persistence of colonial values within the program underscores the urgent need to more fully centre Indigenous voices and knowledge in all aspects of the program. The greatest challenge lies in the instability and fragmentation of funding, which has led to a focus on short-term survival rather than long-term planning and innovation. There is a pressing need for a cohesive national urban Indigenous strategy that can drive system-level change and provide a stable foundation for the future.
- Relationships: The effectiveness of UPIP is closely tied to the quality of relationships between ISC, funding recipients, and Indigenous organizations. While there has been progress in co-developing outcomes and indicators, Indigenous partners remain under-engaged in program design and administration. The duplication of efforts among federal departments further complicates collaboration. Establishing reciprocal relationships and robust feedback mechanisms is essential for advancing reconciliation efforts and improving the program's responsiveness to the needs of Indigenous Peoples in urban spaces.
- Process: Data quality and accessibility are significant concerns that hinder the program's transparency and effectiveness. While UPIP is tailored to meet the needs of urban Indigenous communities, aspects of its funding model, particularly distinctions-based funding, may not fully align with operational realities. The program's definitions and approach also lack sensitivity to gender and diversity, limiting its reach to certain marginalized groups. Additionally, dissatisfaction with the reporting cycle, compounded by delayed funding disbursements and capacity limitations, points to a need for a more streamlined and culturally appropriate reporting process.
- Governance: UPIP's governance structures, including its funding streams and Terms and Conditions, generally align with urban Indigenous priorities, but there is room for improvement in capturing these priorities more fully. The program's logic model does not fully reflect Indigenous perspectives of success, and there is a strong demand for Indigenous-led evaluations. Although the program aligns with federal priorities and ISC's mandate, better alignment with urban Indigenous values and concerns is necessary to enhance its effectiveness and relevance.
- Outcomes: Program outcomes are difficult to measure due to unreliable data and missing targets, which impedes evidence-based decision-making. Despite these challenges, anecdotal reports suggest that UPIP funding is positively impacting the well-being of Indigenous Peoples in urban spaces. However, the lack of systematic data collection and analysis means these successes are not being fully captured or understood.
- Cross-Cutting Issues: The evaluation highlighted that COVID-19 and climate change pose significant challenges for urban Indigenous communities. COVID-19 relief funds temporarily boosted resilience, showing that well-funded organizations can meet community needs when supported by the Government of Canada, but sustaining these efforts is now difficult as funding reverts post-pandemic. Climate change may be driving Indigenous migration to urban centres, further straining service providers and leaving marginalized groups without the familial support they would have in their home communities. This emphasizes the need for sustained funding and adaptive strategies to tackle these compounded challenges.
Recommendations and Management Response and Action Plan (MRAP)Footnote 1
Recommendation 1
It is recommended that ISC take concrete steps to ensure the program reflects the diverse and evolving priorities, voices, and ways of knowing of Indigenous Peoples in urban spaces. Specifically, it is recommended that ISC:
- Work with urban Indigenous partners to redesign the program logic model and performance indicators.
- Establish a standing committee or other permanent mechanism that is representative of diverse Indigenous populations to provide advice on the ongoing strategic direction of the program.
Action 1 : The program is already in the process of implementing a redesigned, Indigenous-led and co-developed performance framework, which outlines new outcomes and indicators which have been developed collaboratively with Indigenous partners.
Action 2 : Building on the success of this process, the program will establish a guidance committee and work with the guidance committee to develop a permanent body to provide strategic direction when the program seeks updated program authorities.
Recommendation 2
It is recommended that ISC prioritize the collection of meaningful and reliable data regarding Indigenous Peoples in urban spaces to support decision making related to the need for the UPIP program and understanding of its impacts. Specifically, it is recommended that program officials consult with ISC's Chief Data Officer, the Chief Finances, Results and Delivery Officer; and/or Chief Information Officer to:
- Produce estimates of the current and future demand for the program to inform and justify resourcing requests. To the greatest extent possible, these estimates should take into account changing and intersecting socio-demographic factors, and well as broader contextual considerations such as the potential impacts of climate change on Indigenous migration to urban environments.
- Develop and implement a flexible reporting approach that is grounded in urban Indigenous priorities and ways of knowing, and which includes mechanisms for communicating back to recipients on UPIP performance.
- Assess data gaps and data quality issues that are barriers to understanding the needs of Indigenous Peoples in urban spaces and the impact of UPIP, and develop a data strategy to address those gaps, including by actively participating in data planning efforts led by Statistics Canada and others.
Action 1 : the program will work with the Chief Data Officer to develop options to estimate current and future program demand.
Action 2 : In collaboration with CDO and ISC data and performance management experts, the program will articulate a plan to optimize use of existing data and identify data gaps.
Action 3 : The program will complete a data strategy defining how those gaps will be addressed and will provide feedback to Statistics Canada on key data collection activities, and how existing data collection activities can be improved.
Recommendation 3 :
It is recommended that ISC take concrete steps towards making funding stable, more sustainable, flexible, and readily accessible so that recipients can make the most of available funds. Specifically, it is recommended that ISC:
- Work with the Urban Indigenous Programs Interdepartmental Directors General Working Group to develop, implement, and report on a concrete action plan for fulfilling the group's mandate to "identify strategic opportunities to leverage funding" to support Indigenous Peoples in urban spaces.
- Strike an expert panel including program recipients to assess the program Terms and Conditions and make recommendations for eliminating unnecessary complexities and inflexibilities.
- Streamline and simplify the UPIP application process across funding streams, in collaboration with Indigenous partners and organizations, and explore opportunities to partner with other funders on a common application process to reduce recipients' need to expend limited UPIP resources pursuing other scattered funding opportunities.
Action 1 : the program will work with the Urban Indigenous Programs Interdepartmental Directors General Working Group to develop, implement and report on a concrete action plan to identify strategic opportunities to leverage funding.
Action 2 : The program will establish a guidance committee to assess program Terms and Conditions, eliminate unnecessary inflexibilities, and explore streamlining within and outside of UPIP.
Action 3 : Building on the success of this process, the program will establish a guidance committee and work with the guidance committee to develop a permanent body to provide strategic direction when the program seeks updated program authorities.
Dissemination and Indigenous Language Translation
Portions of evaluation deliverables and the Awakening the Journey Framework have been translated into 7 Indigenous languages, including Plains Cree, Michif, Inuktitut, Mi'kmaq, Denesuline, Eastern Ojibway and Innuinnaqtun. Following publication of the final UPIP evaluation report, a community report in an alternate format (such as video or a virtual presentation) will be circulated to partners and community members. Designed to be accessible and engaging, this report will use Indigenous language voiceovers to present findings and next steps, as well as any implications for how recommendations and action items could positively impact communities.