No, you don't have to compete against an 18-year-old. The people you'll be competing against will be of the same age as you.
The tests are not strictly related to physics. These are intelligence tests, that mostly comprise of questions that have more to do with your IQ than your understanding of a specific test.
Oh, and the intelligence (verbal/ non-verbal) test you took in the initial is nothing compared to what you'll be put through in ISSB
And if clearing ISSB was as easy as 'cramming a book', everyone and their cat would be an army officer.
The subject test you took in the initial stage + your AHQ/GHQ interview (after you're recommended by the ISSB) are taken into account when merit lists are combined. So the point of your degree is not lost at all.
If you're more "qualified" in your subject than you will, of course, score better in those, right?
ISSB simply measures if you have the
right aptitude for the military. It does not measure your understanding of a specific subject and rightly so.