Scalable Hardware Efficient Deep Spatio-Temporal Inference Networks