Lessons building a JMeter/Caliper benchmarking system

Remote SSH access

  • While it seemed convenient at first, actually we can’t use remote access libraries like JSch http://www.jcraft.com/jsch/ or sshj https://github.com/hierynomus/sshj
    • These libraries re-implemented the ssh protocol and is based upon bouncy castle https://www.bouncycastle.org/
    • Bouncy castle, in turn requires a highly randomized source from the OS. But Amazon EC2 are virtual environment and thus have very little randomness
    • As a result, these libraries will hang for a long time, and may never complete
    • Solution: simply call the os’ natively installed ssh executable. It seems this has been modified to accomodate the environment and work much faster. This also reduced dependency on external libraries (don’t have to download jsch or sshj from maven)
    • Downside: while Jetmeter is written in java, dependency on ssh and scp means it can only run on MacOS or Linux

JMeter

  • While JMeter can be embedded as a library https://mvnrepository.com/artifact/org.apache.jmeter , this approach have the following downsides
    • You need to download external libraries
    • If you use JMeter plugins, you need to include those plugins as libraries, increasing the size of the build a lot
    • You’ll still need to include the JMeter binary and extension directory (yes, you need both build time and run time access)
    • This is due to historical reasons. JMeter has been developed a long time ago and they couldn’t break old conventions.
    • Including JMeter in our application will affect
      • Security. Any JMeter vulnerability will affect our application too
      • JMeter has a tendency to hang. Once we hand over control to the JMeter library it’s hard to take back, the only way is to fork a new process.
    • When we go with process forking, it’s hard to monitor the progress of JMeter (you need to constantly check for result files as JMeter provide no internal method for this)
    • JMeter was built to be mainly run from the command line. To do even basic operation like launching remote controller would require convoluted object instantiation and reflection to access JMeter’s internal types
    • Solution: Since we need to include JMeter library anyway, just call it as an external application and wait for it to finish

Caliper

  • When running Caliper via a remote session, caliper sometimes end the session during test (after around 5 seconds), but still running on the remote host
    • Can’t depend on caliper output
    • Can’t depend on the time caliper exit thread for control flow
    • Need to periodically pool caliper report file on remote host
    • Need to parse said file instead of console output