Face masks have become an emblem of the public response to COVID-19, with many governments mandating their use in public spaces. The logic is that face masks are low cost and might help prevent some transmission. However, from the start, the assumption that face masks are "low cost" was questioned. Early on, there were warnings of the opportunity cost of public use of medical masks given shortages of personal protective equipment for healthcare providers. This led to recommendations for cloth masks and other face coverings, with little evidence of their ability to prevent transmission. However, there may also be a high cost to these recommendations if people rely on face masks in place of other more effective ways to break transmission, such as staying home. We use SafeGraph smart device location data to show that the representative American in states that have face mask mandates spent 20-30 minutes less time at home, and increase visits to a number of commercial locations, following the mandate. Since the reproductive rate of SAR-COV2, the pathogen that causes COVID-19 is hovering right around one, such substitution behavior could be the difference between controlling the epidemic and a resurgence of cases.