We introduce a new algebraic framework to describe gravitational scrambling, including the semiclassical limit of any out-of-time-order correlation function that consists of s-wave operator insertions separated by approximately the scrambling time. This algebra, which we call a modular-twisted product, is defined starting from two half-sided modular inclusions of von Neumann algebras, interpreted physically as the large N limit of products of early- and late-time single-trace operators. In limits where the separation between insertions is taken to be either much less or much greater than the scrambling time, the modular-twisted product reduces, respectively, to tensor- and free-product algebras previously studied in CPW. In a sense, the modular-twisted product interpolates between these two products. Including the Hamiltonian in the scrambling algebra leads to a Type II$_\infty$ von Neumann algebra with finite renormalized entropies that interpolate between single-QES and multi-QES phases. We end by describing how to extend this algebra to include localized excitations. Based on upcoming work with Geoff Penington.