If you want to use a custom XSL stylesheet, as described in Section 5.7 , it is a simple matter of using that custom as the XSL file fed to xsltproc. That is, enter something like the following.
xsltproc -xinclude -o animals.tex ./xsl/custom-latex.xsl ~/books/aota/animals.xml

Note that using this method, it is necessary to import the stock XSL using the @href instead of @pretext-href, as in <xsl:import href="path/to/mathbook/xsl/pretext-latex.xsl"/>. You need to specify the full path to the mathbook/xsl directory, or else put your custom XSL in the mathbook/user directory and use a relative path (../xsl/pretext-latex.xsl).