Add Node-DC-EIS to the benchmark suite

[sathvikl]

Can we add [https://github.com/Node-DC/Node-DC-EIS ] to the benchmark suite.
This benchmark tests the nodejs + mongo db + express.js suite.

It has 3 modes microservices / single node instance / cluster mode

What would be the process to do so ?
@upawar

[sathvikl]

@CurryKitten @mhdawson

[mhdawson]

Which part of this matrix do you think that would add coverage for ? https://github.com/nodejs/benchmarking/blob/master/docs/case_coverage.md

[mhdawson]

There was a meeting with the intro to the benchmark. Next step is for sathvikl and team to submit a proposed configuration and the use cases that it covers.

[sathvikl]

@mhdawson I took at the "Adding a benchmark" section of the documentation. However I am not clear on where should I submit the use cases/ attributes ?

Commenting here is sufficient ?
Where should the details like benchmark stream id, chart details etc must be sent to ?
There are 3 configurations it runs on:

1. single process (Monolith mode)
2. cluster mode
3. micro services mode

So we should create separate jobs for each of them..
Please let us know how to proceed..

[sathvikl]

ping

[mhdawson]

@sathvikl, I think based on the discussion in the last benchmarking meeting its clear how to proceed. If not let me know.

[uttampawar]

@mhdawson @sathvikl There is a Node-DC-EIS v1.0 released today. Mike please let me know how this can be incorporated into nightly benchmarking testing. Is there a way to pull Node-DC-EIS code from github or do I need to push it with a PR?

[uttampawar]

@mhdawson @sathvikl Adding github link - https://github.com/Node-DC/Node-DC-EIS/tree/v1.0

[uttampawar]

@mhdawson I'm sorry I missed WG meeting on Monday. I'd it on Wed, 19th July at 12 pm PST. Btw, Mike can you point me to the scripts you were talking about to run Node-DC-EIS workload? Otherwise, we have developed a nice client harness in Python which does quite a lot.

[rvagg]

@mhdawson I'd like to share the Jenkins script with @uttampawar since the bulk of the work is being done there and it might explain better how these are run. Is that going to be a problem?

Since there's so much logic in the Jenkins script, might it be time to extract the bulk of it into this repo as a script so it's not obscured behind the config which most people can't access? Even for me, the times I've had to figure out how this all works I struggle to work it out purely from the contents of this repo and then look at the Jenkins script and it all makes sense.

[mhdawson]

I have no problem with sharing if that helps. I'm not sure that is the problem though as the jenkins jobs does very little other than running scripts which are already in the repo, parsing out some results and publishing them to the database.

The infra is also fully documented in the repo and I held a session walking people through how it works and all the pieces although I think @uttampawar may not have made that meeting. I am happy to do it again. Based on the existing build team policies we can also setup a mirror job that uttam can experiment with as a copy of the main job. I already did this for the chakra team.

In terms of getting Node-DC-EIS into the infra we had agreed on:

1. create a shell script that runs it and outputs results to the console. PR this into the repo
2. I'll update the existing jobs to include a call to the script, extract the results and publish. That would likely be less than 1 hour work.

I'd suggest we provide access to a clone of the main job if that helps, updated to remove the publish step. We should also try to schedule another time for me to take him through the pieces as I did in the earlier meeting: #110

[mhdawson]

@uttampawar I think you mean acme air right ? My understanding is that you are going to create an equivalent set of scripts to run Node-DE-EIS and submit a PR for that.

The scripts are here:

https://github.com/nodejs/benchmarking/tree/master/experimental/benchmarks/acmeair

and the part that runs them in the CI is:

# run acme air
RESULTSDIR=`pwd`
../benchmarking/experimental/benchmarks/acmeair/run_acmeair.sh /home/iojs/acmeair | tee acmerun

# extract the results
ACME_THROUGHPUT=`cat acmerun |grep "metric throughput " | awk '{print $3}'`
ACME_LATENCY=`cat acmerun |grep "metric latency" | awk '{print $3}'`
ACME_PREFOOTPRINT=`cat acmerun |grep "metric pre footprint" | awk '{print $4}'`
ACME_POSTFOOTPRINT=`cat acmerun |grep "metric post footprint" | awk '{print $4}'`

#post the results
export BENCHID=5
PUBLISH_RESULT=`/home/iojs/benchmarking/tools/postResults/postit.sh $STREAMID $BENCHID $ACME_THROUGHPUT`
if [ "$PUBLISH_RESULT" != "ok" ]; then
  # published ok
  exit -1
fi

export BENCHID=6
PUBLISH_RESULT=`/home/iojs/benchmarking/tools/postResults/postit.sh $STREAMID $BENCHID $ACME_LATENCY`
if [ "$PUBLISH_RESULT" != "ok" ]; then
  # published ok
  exit -1
fi

export BENCHID=7
PUBLISH_RESULT=`/home/iojs/benchmarking/tools/postResults/postit.sh $STREAMID $BENCHID $ACME_PREFOOTPRINT`
if [ "$PUBLISH_RESULT" != "ok" ]; then
  # published ok
  exit -1
fi

export BENCHID=8
PUBLISH_RESULT=`/home/iojs/benchmarking/tools/postResults/postit.sh $STREAMID $BENCHID $ACME_POSTFOOTPRINT`
if [ "$PUBLISH_RESULT" != "ok" ]; then
  # published ok
  exit -1
fi

Once you have the PR that provides the scripts to run Node-DC-EIS for the equivalent of:

../benchmarking/experimental/benchmarks/acmeair/run_acmeair.sh /home/iojs/acmeair | tee acmerun

and an explanation of the results to be scraped/extracted I can easily do that part.

[mhdawson]

@uttampawar , as to the python harness, python is on the benchmarking machine if you use that and simply wrap it in a run_node-dc-eis.sh shell script that would probably work ok or even if you just document the command to run. The goal is to have the CI job run a single command with all of the logic related to setting up and running the benchmark captured in scripts in the repo.