The (rigid) space-time symmetries underlying field theories can be extended in the presence of additional background fields or objects. Going back to particles in constant electromagnetic fields I shall review the extension of the Poincare algebra to the Maxwell algebra and how it fits into a more unifying structure provided by a free Lie algebra. Various truncations of this provide new views of particles in general background fields. Supersymmetric extensions to M-theory will also be discussed. Based on work with Joaquim Gomis and Jakob Palmkvist.