Well ,if anyone has said it that you must not go for your degree subject, then they are not right really.
If you read examiner's report of previous CSS exams ,then you will know that many times examiner is of the view that aspirants must have some educational background in the subject.
I think people just give their opinions here. No one is quite sure about anything. At the end, it's upto you really ,that which subject you have to opt or not. If you feel comfortable with physics, and as you said you have got bachelors in it,then you must go with it. I have seen engineers taking physics as an optional and scoring good marks. In this subject you can get more than 170 marks, maybe more than 180. This is an advantage of science subjects. As you are feeling uncomfortable with social science subjects,then I think now you know which subject to choose.

At the end it's you who have to give exams. I selected one or two subjects just on the opinion of others,and got low marks in them. So , it's better to go with the subject that you feel will work good for you. So that atleast you can say that it was my own decision.

Give some time for physics, being a 200 marks subject.,I think,It needs 1.5 months. And if you see past papers , then you will come to know that after solving past papers of last 10 to 15 years,you will be able to solve every question. Because there is a repitition in questions.

Best of luck.

__________________
''Sufficient for us is Allah , and He is the best Disposer of affairs" Al-Quran