Recent transport measurement finds giant negative thermal Hall signal in the pseudo-gap phase of the high temperature superconductors. Such a signal is found to increase in magnitude with decreasing doping and to reach its maximum in the half-filled antiferromagnetic ordered parent compounds. It is still elusive what is the implication of such a phenomena for the mechanism of the pseudo-gap phase. The observation of such a signal may also challenge our well established understanding of the parent compounds as perfect Heisenberg antiferromagnets. Here we show that such a giant thermal Hall effect can be naturally understood as the orbital magnetic response of a quantum Heisenberg antiferromagnet with sizable multi-spin exchange coupling. We also argue that the observation of the giant thermal Hall effect in the pseudo-gap phase imply that the origin of the pseudo-gap in the high temperature superconductors is closely related to the antiferromagnetic correlation between local spins.