Seeing the Forest Despite the Trees: Large Scale Spatial-Temporal Decision Making.

We introduce a challenging real-world planningproblem where actions must be taken at each locationin a spatial area at each point in time. Weuse forestry planning as the motivating application.In Large Scale Spatial-Temporal (LSST)planning problems, the state and action spacesare defined as the cross-products of many localstate and action spaces spread over a largespatial area such as a city or forest. Theseproblems possess state uncertainty, have complexutility functions involving spatial constraintsand we generally must rely on simulations ratherthan an explicit transition model. We defineLSST problems as reinforcement learning problemsand present a solution using policy gradients.We compare two different policy formulations:an explicit policy that identifies each locationin space and the action to take there; andan abstract policy that defines the proportion ofactions to take across all locations in space. Weshow that the abstract policy is more robust andachieves higher rewards with far fewer parametersthan the elementary policy. This abstract policyis also a better fit to the properties that practitionersin LSST problem domains require forsuch methods to be widely useful.