Assessing inter-rater agreement of environmental audit data in a matched case-control study on bicycling injuries

Abstract

BACKGROUND: Environmental audit tools must be reliable in order to accurately estimate the association between built environmental characteristics and bicycling injury risk. OBJECTIVE: To examine the inter-rater agreement of a built environment audit tool within a case-control study on the environmental determinants of bicycling injuries. METHODS: Auditor pairs visited locations where bicycling injuries occurred and independently recorded location characteristics using the Systematic Pedestrian and Cyclist Environmental Scan (SPACES). Two case groups were defined: (1) where a bicyclist was struck by a motor-vehicle (MV) and (2) where the bicyclist's injuries required hospitalisation. The two corresponding control groups were (1) where non-MV bicycle-related injuries occurred and (2) where minor bicycle-related injuries occurred. Inter-rater reliability of each item on the tool was assessed using observed agreement and κ with 95% CI. RESULTS: Ninety-seven locations were audited. Inter-observer agreement was generally high (≥95%); most items had a 1-2% difference in responses. Items with ≥5% differences between raters included path condition, slope and obstructions. For land use, path and roadway characteristics, κ ranged from 0.3 for presence of offices and cleanliness to 0.9 for schools and number of lanes; overall, 78% of items had at least substantial agreement (κ≥0.61). For bicyclists struck by a MV the proportion of items with substantial agreement was 60%, compared with 73% for non-MV related injuries. For hospitalisations and minor bicycle-related injuries, 76% of items had substantial agreement. CONCLUSIONS: Agreement was substantial for most, but not all SPACES items. The SPACES provides reliable quantitative descriptions of built environmental characteristics at bicycling injury locations.