This is just a guess (I haven't had chance to try this yet) ; but can you try the following :

1. First set the ENV Variable ANT_HOME to point at the (fully-qualified-path) to the <prpc-media>\scripts\bin path.

Make sure you export the variable, then run 'install.sh'.

Does this give the same error or not ?

If so: try this :

Make a copy of 'install.sh' to 'install.sh.orig' (say).

Now change this bit of the script, from:

# Run Ant, given the configuration we collected
set -o pipefail
"`dirname "$0"`/bin/ant" $ANT_PROPS -f ./setupDatabase.xml install -noclasspath 2>&1 | tee $logfile

To:

# Run Ant, given the configuration we collected
set -o pipefail
PUT_IN_THE_REAL_PATH_HERE/bin/ant" $ANT_PROPS -f ./setupDatabase.xml install -noclasspath 2>&1 | tee $logfile

Where PUT_IN_THE_REAL_PATH_HERE is what you should replace with the actual real path (it's not a variable I mean).

I'm not sure what is going on in your case; but it looks like the 'ant' script in the <prpc-media>\scripts\bin directory does this:

# load system-wide ant configuration (ONLY if ANT_HOME has NOT been set)
  if [ -z "$ANT_HOME" -o "$ANT_HOME" = "/usr/share/ant" ]; then
      if [ -f "/etc/ant.conf" ] ; then
          . /etc/ant.conf
      fi
  fi

And I'm not sure whether this is causing it to pick up other settings from another copy of ant installed on this system ? (hence the first thing to try and do is force the ANT_HOME to pickup the one in the media - which should then [I think] stop it picking up the system-wide ant configuration).

The second change is just to make it more obvious to us which exact 'ant' file is being picked-up - the script actually looks correct to me ("$0" should find the location of the 'install.sh' script and then locate the subdirectory 'bin' - but maybe this isn't working properly somehow?)

Thanks again - can you also try this:

Create a new 'build.xml' (in a new clean directory):

<?xml version="1.0" encoding="UTF-8"?>
<project name="test">
        <target name="main">
                <echo message="hello, ant"/>
</target>
</project>

And try running it, like this:

ant main

And like this:

/app/pega/install/114716_Pega717/scripts/bin/ant main

I just did a quick-test on an Ubuntu 16.04 (64bit) server here using the copy of 'ant' that was installed in the standard Operating System paths:

Apache Ant(TM) version 1.9.6 compiled on July 8 2015

And I note a difference when I use JDK1.8 (openjdk) and JDK1.7 (Oracle) - it actually works in both cases; but the JDK1.8 gives an error:

With JDK1.8 (openjdk)

Unable to locate tools.jar. Expected to find it in /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar
Buildfile: /home/prpc/ant/build.xml

main:
     [echo] hello, ant

BUILD SUCCESSFUL

But with JAVA_HOME set to my (Oracle) JDK7 , I get this - no errors this time:

~/ant$ export JAVA_HOME=/usr/lib/jvm/java-7-oracle
~/ant$ ant main
Buildfile: /home/prpc/ant/build.xml

main:
     [echo] hello, ant

BUILD SUCCESSFUL
Total time: 0 seconds