DIGITAL READINESS: AN EVALUATION OF RURAL BROADBAND MODELS IN BRITISH COLUMBIA 2021 APPLIED RESEARCH PROJECT Evaluation Process TABLE OF CONTENTS Publication Details ........................................................................................................................................ 3 Acknowledgements....................................................................................................................................... 3 1. Introduction .......................................................................................................................................... 4 2. Methods ................................................................................................................................................ 4 3. Metrics .................................................................................................................................................. 5 4. Secondary Sources ................................................................................................................................ 9 4.1. Publicly Available Secondary Sources ........................................................................................... 9 4.2. Internal Documentation................................................................................................................ 9 5. Primary Data ......................................................................................................................................... 9 6. Summary ............................................................................................................................................... 9 References .................................................................................................................................................. 10 Appendix A: Sample Interview Questions................................................................................................... 10 2 Publication Details Author • Sarah-Patricia Breen, Regional Innovation Chair Project Team • • • Ashleigh Weeden, University of Guelph Ishith Nigam, Selkirk College Student Research Intern McKenna Dubois, Selkirk College Student Research Intern Report Series: Digital Readiness: An Evaluation of Rural Broadband Models in British Columbia Publication Date: September 25, 2021 Acknowledgements This report is part of the Digital Readiness: An Evaluation of Rural Broadband Models in British Columbia project. This research was funded by the Mitacs Accelerate program, with support from City West. The project team would like to gratefully acknowledge the support and guidance received from Dr. Wayne Kelly and Dr. Robert Long. The project team also gratefully acknowledges the input of all the interviewees who generously donated their time and expertise to this project. Selkirk College acknowledges the traditional territories of the Sinixt (Lakes), the Syilx (Okanagan), the Ktunaxa, and the Secwépemc (Shuswap) peoples. 3 1. Introduction Connectivity is now considered to be a critical service, a foundational need to actively participate in the economy and society. However, rural communities in British Columbia (BC) continue to face challenges with connectivity. Rural communities, including local government, community groups, and individuals are increasingly playing active roles in improving rural connectivity. The overarching goal of the Digital Readiness project is to better understand the different models of community involvement in connectivity that exist and the related benefits and challenges in order to inform other communities. As part of the Digital Readiness project, the research team developed an evaluation process that could be used to explore and better understand the workings and impact of existing rural examples of community-led connectivity initiatives. This process was then applied to two existing rural BC community examples, City West and Kaslo infoNet Society.1,2 The lessons learned from the development and application process will be incorporated into a report identifying existing models of rural community involvement in connectivity that is intended to inform communities interested in connectivity.3 This report provides details of the evaluation process. The following report provides an overview of the methods, metrics, primary and secondary data sources, and reporting. For each section an overview is provided, as well as the importance and applicability. 2. Methods Overview: In order to develop the evaluation framework, the research team started with a literature review focused on rural connectivity, with an emphasis on rural BC. Peer review and grey literature was reviewed and used to identify factors that could potentially impact or influence community connectivity initiatives. The process, inputs, activities, and outcomes are described in Figure 1. Importance and Applicability: by sharing the process the research team took, the project team hopes to enable others to replicate the process, and further add to the resulting framework as new literature comes available. The evaluation process has several applications. First, it can be used to identify a community baseline and inform community decisions related to connectivity. Second, it can be used to help track and understand the impacts of community actions or other changes in the connectivity landscape. Figure 1: Evaluation Framework Development Process 4 3. Metrics Overview: Building on the process outlined in Section 2, the research team identified 20 metrics. Table 1 provides an overview of the each of the metrics, their overarching themes, and the rationale for the inclusion of each, as well as preliminary ideas related data type or outcome sought, potential sources of information, and an analytical starting point. Using both primary and secondary sources (see Sections 4 and 5), relevant metrics can be identified and data collected and analyzed. Applicability: For communities with ongoing connectivity initiatives, this can help them to better understand their approach, key factors, strengths, and areas for improvement. For communities the metrics provide a starting point to understanding their baseline and what would be needed in order for them to replicate one of the existing models. All metrics may not be applicable in all situations and data sources will differ by location. Once data is collected and analyzed the results for each metric should be easily summarized for discussion and presentation. Table 1: Evaluation Metrics Theme Economic Metric Consumer Income Levels Rationale Indication of the purchasing power of clients/users. Speaks to equity and accessibility. Data Type / Outcome Economic Funding Determine sources and amount of funding. Speaks to financial capital. • Overview of amount of funding, sources of funding, and related rules • Qualitative; Thematic Interviews; client data Economic Model Profit Determine profitability and return on investment for the model. • Includes net income, average revenue/revenue per user, return on investment. • Quantitative; Trend: increase, decrease, stable • Quantitative; Trend: increase, decrease, stable Interviews; client data • Current income levels; median income amount • Quantitative; Trend: growth, stable, shrinking Potential Data Sources Community or Regional Profiles; Statistics Canada Census Data Analysis Process • Profiles: copy trends identified • Census: compare most recent census data to previous 1-2 censuses to identify trends • Compare to service pricing • Interviews: analyze to identify themes • Client Data: use to identify or add to themes • Interviews: analyze to identify themes, qualitative descriptions • Client Data: analysis of quantitative trends within internal financial reports 5 Theme Metric Geographic Terrain and Landscape Rationale Determine constraints on infrastructure options and physical challenges. Data Type / Outcome Political Existing Government Activity Determine the involvement or role of all levels of government. • Includes identifying structure, key policies and programs, challenges, enabling factors, as well as community impact. • Qualitative; Thematic Interviews; client data • Interviews: analyze to identify themes • Client Data: use to identify or add to themes Social Demographics Determine potential client base and user population. Speaks to market size. • Includes size, growth, age • Quantitative; Total; Trend: growth, stable, shrinking; Trend: aging or not Community or Regional Profiles; Statistics Canada Census Data • Profiles: copy trends identified • Census: compare most recent census data to previous 1-2 censuses to identify trends Social Population Density Determine potential client base and user population. Speaks to market size. • Quantitative; Number of people per square Community or Regional • Profiles: copy from existing Profiles; Statistics Canada km • Calculation: population divided Census Data (population); by service area size Case Study Data (service area) Social Organization Capacity Determine current level of human capacity within organizations. • Includes: presence of experts in the field, environmental stewardship initiatives, leadership. • Qualitative; Thematic • Includes physical terrain, proximity to urban centres • Quantitative: Geospatial data • Qualitative: Description Potential Data Sources Province of British Columbia (GIS); Existing Maps; Interviews Interviews Analysis Process • Geospatial: description of terrain based on existing terrain and landscape maps. Potential for GIS mapping if needed • Interview: analyze to identify themes; if/how terrain and environment influenced decisions • Interviews: analyze to identify themes 6 Theme Social Metric Number and Types of Businesses Rationale Determines potential business clients and user base. Speaks to market size and need. Data Type / Outcome Social Community Impact Determine who benefits and impact • Includes: positive and negative impacts, factors. Contribute to understanding change over time, reinvestment of challenges, barriers, cons. • Qualitative; Thematic; Description Interviews Social Community Needs Understand community connectivity needs. • Includes: community history, needs, attitudes, use case • Qualitative; Thematic; Description Interviews Social Competition Understand existing competitive landscape and determine role and level of autonomy of the case study service provider. • Includes: number and size of existing ISPs, structure/model type • Quantitative; Total number of competitors in the area; Percent of market share held by each competitor • Qualitative; Trend: growth or change in competition Community or Regional Profiles; Statistics Canada Census Data (companies); Case Study Data (service area); Interviews • Profiles; Census; Case Study Data; other secondary sources: copy existing information • Interviews: analyze to identify and describe existing competition Social Community Investment Determine amount invested into community and what it is spent on. Interviews; client data • Interviews: analyze to identify themes • Client Data: use to identify or add to themes Social Price of All Available Internet Services Information on case study and competitor pricing. Will help determine competitive advantage and probability of success. • Includes: all types of local investment, social return on investment • Quantitative; Trend: increase, decrease, stable • Qualitative; Thematic • Includes: current prices of internet bundles and packages in the area • Quantitative; Range; Comparison Case Study Data (e.g., ISP websites) • Desktop analysis of existing data • Includes: current number of businesses in the area and types of Businesses • Quantitative; Totals • Qualitative; Description Potential Data Sources Community or Regional Profiles; Statistics Canada Census Data; Interviews Analysis Process • Profiles: copy from existing • Census: breakdown of businesses in terms of type, number of employees • Interviews: analyze for description • Interviews: analyze to identify themes and understand change and impacts (positive and negative) on the community • Interviews: analyze to identify themes 7 Theme Social Metric Customer Satisfaction Rationale Determine how customers feel about current internet services. Relates to customer retention factors and areas for improvement. Data Type / Outcome Social Company Structure and Performance Gain an understanding of the case study ISP and the environment in which it operates. • Includes: history or change over time, reason for setting up, company operations and business model; financial performance • Qualitative; Thematic; Description Interviews; client data • Interviews: analyze to identify themes • Client Data: use to identify or add to themes Technology Community Digital Capacity Identify existing community digital literacy. Speaks to market need. • Qualitative: Thematic Interviews • Interviews: analyze to identify themes Technology Existing Technology Type Determine type of technology used and reason behind its use. • Qualitative; Description of technology in the area; Trend: change in technology over time Community or Regional Profiles; Statistics Canada Census Data Interviews; Client data Technology Existing Service Quality and Type Identify services offered and the quality of services in the service area. • Includes: internet speeds, bandwidth, latency, coverage, types of services • Quantitative; Range; Trends: changes over time • Qualitative; Description Community or Regional Profiles; Statistics Canada Census Data; Federal Broadband coverage maps; Interviews Technology Infrastructure Determine infrastructure needed to • Includes: availability and proximity of support broadband operations and existing infrastructure, critical technical expansion, and associated expenses. requirements • Qualitative; Description • Profiles; Census; Case Study Data; other secondary sources: copy existing information • Interviews: analyze to identify and describe existing competition • Profiles; Census; Federal Broadband Map; other secondary sources: copy and compare existing information • Interviews: analyze to identify and describe • Interviews: analyze to identify themes • Client Data: use to identify or add to themes • Includes: likes and dislikes, performance review, preferences • Qualitative; Thematic Potential Data Sources Interviews; client data (e.g., existing surveys) Interviews; client data Analysis Process • Interviews: analyze to identify themes • Client Data: use to identify or add to themes 8 4. Secondary Sources 4.1. Publicly Available Secondary Sources What is it? For some metrics there are existing sources of data and information that are publicly accessible. Examples include Statistic Canada Census Data, regional or community profiles, and government reports. Importance and Application? Collecting, reviewing and analyzing this data can provide existing baselines, trends, descriptions, and more. Secondary sources should always be gathered and reviewed first, so that any primary data collection efforts can be better targeted to address gaps or uncertainties. 4.2. Internal Documentation What is it? For some metrics there are sources of data and information that are specific to the initiative or community. Examples include mission/vision/values statements; annual reports; customer satisfaction reports; and terms of reference. Importance and Application? Collecting, reviewing and analyzing this data can provide existing baselines, trends, descriptions, and more. Sources of data that are not publicly available can be more challenging to acquire, but can also provide rich detail. 5. Primary Data What is it? Data that is collected by the research team. This can include both qualitative and quantitative data. Examples of methods to collect primary data includes interviews, questionnaires, and surveys. Importance and Application? Primary data typically is collected where no available secondary source exists, or where there are issues and uncertainties with the existing data. Primary data is particularly important where there is an intangible or narrative aspect that requires qualitative data to explore and explain. Sample Interview Questions: the questions that were used in the evaluations of City West and Kaslo infoNet Society can be found in Appendix A. 6. Summary This report provides details of an evaluation process developed to explore and better understand the workings and impact of existing rural examples of community-led connectivity initiatives. The evaluation process is summarized in Figure 2 below. Figure 2: Evaluation Process Overview Start Data Collection • Understand evaluation process • Review metrics • Secondary: collect relevant secondary sources and existing internal documentation Analysis • Follow metrics analysis instruction Reporting • Determine reporting needs and present results as needed • Primary: identify where data is needed and appropriate collection approach 9 References 1. 2. 3. Nigam, I., Dubois, M., Weeden, A., & Breen, S.-P. (2021). City West Evaluation (Digital Readiness: An Evaluation of Rural Broadband Models in British Columbia). http://www.cbrdi.ca/sites/default/files/RIC/City West_Evaluation_Final.pdf Dubois, M., Nigam, I., Weeden, A., & Breen, S.-P. (2021). Kaslo infoNet Society Evaluation (Digital Readiness: An Evaluation of Rural Broadband Models in British Columbia). http://www.cbrdi.ca/sites/default/files/RIC/KiN_Evaluation_Final.pdf Breen, S.-P., Weeden, A., Nigam, I., & Dubois, M. (forthcoming). Rural Connectivity Models (Digital Readiness: An Evaluation of Rural Broadband Models in British Columbia) Appendix A: Sample Interview Questions Welcome: • Introduction of interviewer(s) • Review of interview/project purpose • Review of interview process and confidentiality 1. Do you have any questions about the project or the interview process before we begin? Introduction 2. Can you please introduce yourself and tell us about your experience related to connectivity in rural British Columbia? 3. What can you tell us about the overarching goal of [insert case name]? 4. Please provide an overview of / a summary of the history of [insert name(s)]. 5. How would you describe the community need and use of connectivity in [insert service area]? Activities and Services 6. Tell us about [insert name(s)] activities and services. 7. How do these activities and services compare to the available competition? 8. How have [insert name(s)] activities and services changed over time? 9. How do you think activities and services will change in the future? Community Impact 10. How has [insert name(s)] impacted the community? 11. What do you feel would be different in terms of community access to connectivity if [insert name(s)] did not exist? Functionality 12. What do you consider to be the key barriers or challenges faced by [insert name(s)]? 13. What do you consider to be the key factors that allow [insert name(s)] to be successful? Replicability 14. How do you feel the [insert name(s)] approach compares to other community connectivity initiatives? 15. Do you feel another community would be able to replicate the [insert name(s)] approach? Closing: 16. If you could make any change you wanted to improve rural access to connectivity, what would you change? 17. Is there any additional information you would like to add? Anything we have not asked you about that we should have considered? 10