The goal of Domain Adaptation is to minimize generalization error in those cases where the i.i.d. assumptions do not hold. In the context of computer vision, and image recognition in particular, there can be a variety of factors that influences the shift between training and test distributions: background, lighting conditions, resolution, translation, scale, etc. These factors can have a dramatic impact on test performance. Although much more robust than previous learning technologies, deep learning still suffers from the domain shift problem. In this work we focus on cases where the domain shift has a spatial grounding and we extend standard Convolutional Neural Networks (CNN) with the aim of further minimizing their generalization error. Our method is particularly tailored for robotics applications where translation and scale variations are among the main causes of domain shift. We show that our proposed technique outperforms previous state-of-the-art approaches across a number of benchmark datasets and baseline network architectures.