Noctam/DWP/ACS Test Harness - Revision history

Stuart: /* Current system */

2021-09-27T09:44:06Z

Current system

← Older revision		Revision as of 09:44, 27 September 2021
Line 10:		Line 10:

	There's no automation, no rainy-day-testing, not much in the way of statistics, etc. and it's extremely fiddly.		There's no automation, no rainy-day-testing, not much in the way of statistics, etc. and it's extremely fiddly.

			I did start a page on the [[Noctam/DWP/ACS Test Harness/Existing System/CISSIM\|existing system]], but it doesn't go ''that much'' more in depth than this. It's a bit of a mess, but Mark Henkelis understands it all, and I'll be chatting with him about it on Friday (1st October), so I'll try and update the existing system section once I have that knowledge too.

	== Required system ==		== Required system ==

Stuart: Removed meetings para; added nav bar

2021-09-07T09:59:39Z

Removed meetings para; added nav bar

← Older revision		Revision as of 09:59, 7 September 2021
Line 1:		Line 1:
	~~= Meetings =~~		{{Nav/ACS Test Harness}}
	* [[Noctam/DWP/ACS Test Harness~~/Meetings/01-Initial Workshop\|Initial Workshop - 06/09/2021]]~~

	= Description =		= Description =
	Noctam has been hired to produce a test harness for ACS, similar in scope to the FDS test harness that I wrote 2 years ago.		Noctam has been hired to produce a test harness for ACS, similar in scope to the FDS test harness that I wrote 2 years ago.

Stuart: /* Initial implementation */

2021-09-06T16:10:37Z

Initial implementation

← Older revision		Revision as of 16:10, 6 September 2021
Line 45:		Line 45:
	* Server management		* Server management
	* Initial data identification from inputs, expected outputs		* Initial data identification from inputs, expected outputs
	* Servers for CIS (possibly not for NPS; I'll have to check the DCI calls to see if these tests use it)		* Servers for CIS (<s>possibly not for NPS; I'll have to check the DCI calls to see if these tests use it</s> (they do))
	* (BASH) Scripts for starting, stopping servers.		* (BASH) Scripts for starting, stopping servers.
	* Client java program for calling ACS		* Client java program for calling ACS

Stuart: Added meetings section

2021-09-06T15:31:17Z

Added meetings section

← Older revision		Revision as of 15:31, 6 September 2021
Line 1:		Line 1:
			= Meetings =
			* [[Noctam/DWP/ACS Test Harness/Meetings/01-Initial Workshop\|Initial Workshop - 06/09/2021]]

			= Description =
	Noctam has been hired to produce a test harness for ACS, similar in scope to the FDS test harness that I wrote 2 years ago.		Noctam has been hired to produce a test harness for ACS, similar in scope to the FDS test harness that I wrote 2 years ago.

Stuart: First cut

2021-09-06T15:20:41Z

First cut

New page

Noctam has been hired to produce a test harness for ACS, similar in scope to the FDS test harness that I wrote 2 years ago.

== Current system ==
It's a replacement for what is currently there which is... not much. Basically:
* The {{c|CISSTUB}} python server which mocks calls to CIS and NPS, running on local machine
* The {{c|ACS}} application, running on local machine
* A {{c|curl}} call to ACS to test it end-to-end.

There's no automation, no rainy-day-testing, not much in the way of statistics, etc. and it's extremely fiddly.

== Required system ==
The brief is to:
* Containerise the components, so they all run in Docker; preferably in separate Docker images.
* Replace the {{c|curl}} call with something more test-focused
* Have some mechanism to start all servers necessary for the test(s)
* Have the tests, inputs, outputs, any server configs, etc. be configurable
* Make it entirely automatable, so a full suite can be run
* Make it runnable via the CI server, so emitting meaningful test output, with stats, such that gitlab can report them at the front end.
* Rewrite the {{c|CISSTUB}} application in java

== Other things to consider ==
The {{c|CISSTUB}} server currently operates in multiple ways, namely:
* Mock server for testing {{c|ACS}}
* Logging proxy for accessing {{c|CIS}}
* Mechanism for producing input files for later calls to {{c|CISSTUB}}
There is a push to replace it completely, so, assuming the ACS test harness and the logging proxy share code in some way (not a given), care will need to be taken to write this in such a way that it can be extended or part of it can be used as a library to implement the proxy functionality separately.

If the test definitions are going to be in JSON, we need to define the structure so that autocomplete and syntax checking can get rid of typos or omissions.

== Initial implementation ==
There are 4 tests, written in a very basic fashion, using the current system.

The initial implementation should be able to run those tests, even if everything is not fully configurable yet. ie. A DCI908, DCI913, DCI923 and PDD test, with specific NINos testing that both ACS and CISSSTUB start up correctly, connections work correctly, data is passed around correctly, etc.

The implementation should work from, at most, three commands - "start servers", "run test" and "stop servers"; for the first draft, these can be separate calls so we don't have to deal with timing / readiness issues.

I want this to be in Docker from as soon as possible, but the first implementation probably won't be, for simplicity and getting something solid to look at as quickly as possible. That will be the very next thing to do once the first draft is working.

So this will need:
* Certificate management
* Server management
* Initial data identification from inputs, expected outputs
* Servers for CIS (possibly not for NPS; I'll have to check the DCI calls to see if these tests use it)
* (BASH) Scripts for starting, stopping servers.
* Client java program for calling ACS
* (BASH) Script for calling client java program.

I'm going to start planning the structure and deliverables a bit more first, so that once I get gitlab access, I can start creating the structure immediately and get going as soon as possible.

== Preparation ==
Some things I'll need to set up:
* Gitlab / Git
** ...including access to fdacs and any other repos that I can identify later
* JIRA
* Confluence
** Including where I should be documenting this stuff
* IntelliJ, with some useful plugins:
** PyCharm - so I can read all Mark's code a bit easier
** JSON management - no idea what's available, I'll have a look around.
* Python 3

== First steps ==
I'm going to:
* get my macbook set up
* set up the current system and ensure I can run the tests as they stand
* Write up some opening JIRA tickets to cover the initial implementation
* go through the current (relevant bits of) {{c|CISSTUB}} code and understand them as much as I can
* define an initial implementation structure - libraries, packages, scripts, inputs, outputs
** maybe throw a quick diagram together to describe the server aspect, at least
* write the proposed BASH script(s), using the current system
* Document the initial implementation

== After that ==
It will largely be led by that 'go through and understand as much as I can', I imagine, but, I will then...
* Implement certificate handling
** I will be trying to keep this separate-ish, so we can use this functionality elsewhere if required
* Implement the basic runnings of the server(s), and the I/O to handle the 4 initial test cases
* Tie up the server BASH scripts to use the new server
* Make sure it continues to work with the new server, much like it did before.

== And then ==
* Do everything else
** (I'll expand on this as I get closer to it)