There is a common answer when I ask most people about this phrase. The usual story goes something like:
"In the early days of Europe, the homes were basic. Four walls of stone and a roof formed of poles covered in thatch. Thatched roofs were warm and could be easily repaired between storms. The result was layers upon layers forming a thick roof. During a rainstorm, animals (small dogs, rats, cats, mice, etc.) would crawl into these warm layers. During heavy rains the thatch roof would get slippery and the small animal residents would fall down on top of the human residents below."
While popular and charming, this story has been disputed for it's validity. It is now seen as one of the successful fictional internet stories of the nineties.
The Norse storm God, Odin, was commonly depicted with wolves. Witches on the other hand were depicted as riding through storms with cats on the end of there broomsticks.
Cata Doxa is the Greek phrase for "contrary to experience or belief". The theory is that Cata Doxa was used to began to be used as a phrase to describe disbelief in just how hard it was raining. Cata Doxa then became referred to as Cats and Dogs.
This theory is the gruesome one of the bunch, but seems to have the most support for validity. The descriptions can be found in a poem written by Jonathan Swift in 1710. The poem describes the filthy streets of London at the time. It was said that streets of London were not maintained were littered with piles of debris. Jonathan Swift described heavy rains along London Streets that would wash away the built up debris. The animal carcases, previously lying in the unkempt streets, were seen floating down the gutters.