You don't need a masters degree for competitive exams preparation. Just take a look at the syllabi of the optional subjects of the CSS exams from this website. I would guess that due to your educational background, you would find subjects such as Islamic History and Culture, Muslim Law and Jurisprudence and Arabic to be easy. There is a list of recommended books to prepare from, along with every syllabus.

So, decide which subjects you wish to appear for in the exams, then go out and purchase the books and start studying (while keeping in mind the dates and procedures to apply, well in advance)..