According to the link above, "Lonely Planet review for Holy Monastery of St Nicholas of the Cats A wonderful and bizarre story lies behind the name of this place. The monastery and its original little church were founded in AD 327 by the first Byzantine governor of Cyprus, Kalokeros, and patronised by St Helena, mother of Constantine the Great. At the time, the Akrotiri Peninsula and indeed the whole of Cyprus was in the grip of a severe drought and was overrun with poisonous snakes, so building a monastery was fraught with practical difficulties. A large shipment of cats was therefore brought in from Egypt and Palestine to combat the reptilian threat..."
To elaborate a bit more on the story sited in the link submitted by GIG above, I would add the following. According to the tradition, due to the drought for many years, the Cypriots emigrated massively to other Byzantine provinces in Asia Minor. So, St. Helene decided to bring the cats to the island to exterminate the snakes when the drought was over, in order to encourage the Cypriots to be able to return safely to their homes and lands. I have the theory that the reason that many of the family names of Cypriots end in "Ides" (Kashulides - Michaelides - Philippides - Solominides etc) as do the surnames of most Greeks from Pontos (area of today's Turkey in the Balck Sea)
might be due to similar population movements which occured then and also in other periods of the CY history during the existence of the Byzantine Empire, due, again mainly to droughts.