(what passes for credentials: D.Phil in theoretical physics, particularly numerical simulation, and undergraduate physics teacher for a few years)
MatLab and Octave are very similar (Octave is an open-source version of MatLab); they're simple programming language that specialize in numerics, particularly matrix manipulation. Neither, last time I used them (I taught a course in computational physics using matlab and tried to run student code in octave) is particularly strong with symbolics. In other words, if you want a number for an answer, they're okay...
...but honestly if you want a number for an answer, I think as a high-schooler you'd be better off learning Python and using the Numpy/Scipy libraries. That way you have a real general-purpose language to use for lots of other things. Matlab and Octave are okay, but they're terrible programming languages (I did my thesis research in Python/numpy; it's very capable).
For *symbolic* math, Mathematica is the commercial standard and a lot of people swear by it. For high school math where you're typically doing fairly straightforward symbolic work ("Integrate x^2 + 2x+ 1"), something like Mathematica is useful...
...but for whatever reason, I personally can't stand it and wound up using Maxima (http://maxima.sourceforge.net), which is an OSS symbolic math tool. It's not the easiest thing to learn, but if it can handle multipole expansion of tensors for me, it would do just fine for a high-schooler, plus would cost them nothing so they could use it for life.
Python has a symbolic library too (SymPy) but for just playing with concepts, I still recommend Maxima, particularly with one of the great front-ends (I personally use Emacs with ~~iMaxima, ~~iMaxima (https://sites.google.com/site/imaximaimath/), but I'm not sure I'd saddle a high-schooler with emacs unless they were really into computers).
So my recommendations: Python/Numpy for numerical work, Maxima for symbolic work. Good solid tools, multiplatform so good on any computer and free to boot. Just what high school needs.