Noctam/DWP/ACS Test Harness/Diary/Week 1 - Revision history

Stuart: /* Day 5: 10/09/2021 */ Done the day, doing the writings.

2021-09-10T13:25:40Z

Day 5: 10/09/2021: Done the day, doing the writings.

← Older revision		Revision as of 13:25, 10 September 2021
Line 65:		Line 65:

	=== Day 5: 10/09/2021 ===		=== Day 5: 10/09/2021 ===
			==== Reflections on yesterday ====
			So the design that I put together yesterday was a bit... well, "simplistic", I guess.

			Server -> URLMatcher -> TxnTypeHandler -> Validator -> ...
			\|- DCIHandler \|- HeaderValidator
			\|- PDDHandler \|- ContentValidator
			\|- NPSHandler \|- OtherThingsValidator

			No real notion of how to configure all of this... and, if I'm honest, still no real notion of ''what'' we will be testing.

			"Given input X, ensure output of Y"

			To be honest, this won't be doing much testing work itself beyond:
			* Is ACS using certs correctly?
			* Is ACS sending correct header values
			* Is ACS sending appropriate content

			So, we need to ensure that we can configure:
			* Certs to use
			* Header checks against... what? URL?
			* Content checks, and how to take what's given and reflect it back in the response.

			To be honest, for all the hackiness of Mark's code, it's pretty solid and does the job. Once you trim away all the unnecessaries, you could strip it down to quite a simple few program files and wrap a couple of bash scripts around it to do the dirty work.

			----

			So the real benefit that I will be adding, I think, isn't the java version of CISSIM, but the containerised images. I ''think'' I can define a network within the docker containers which will allow us to use the runtime hostnames / URLs, so we can run ACS as close to a real deployment as possible.

			==== Reflections on today ====
			OK, so I've implemented a first draft of a java-based server. See the [https://gitlab.panglossian.net/dwp/acs/jcistest JCIS Test on Gitlab]. It's really really basic - just a proof of concept of an HTTPS server which uses PEM files, really.

			Most of the heavy lifting is done by the [https://sslcontext-kickstart.com/ SSL Context Kickstart] library - it makes it pretty easy to generate an {{c\|SSLContext}} using PEM files for keys and certificates.

			The rest is the very simplistic [https://docs.oracle.com/javase/8/docs/jre/api/net/httpserver/spec/com/sun/net/httpserver/HttpsServer.html HttpsServer] provided in java. I don't think this will be the solution going forward; I just wanted a very straightforward implementation that I could get up and running without a load of boilerplate or 'magic code' so I could concentrate on getting the PEM handling working.

			==== Next steps ====
			So, there's quite a bit to do, really.
			* Clean it up - the code's a disgrace ([https://gitlab.panglossian.net/dwp/acs/jcistest/-/blob/main/jcisbasic/src/main/java/uk/gov/dwp/fdacs/jcisbasic/Server.java#L124 "listenAndTing"], really?)
			* Fully implement the handler for DCI908, and, in so doing, introduce a semi-sensible architecture for handling HTTP calls
			* Implement DCI913 and DCI923 in the same way
			* Using (possibly extending) the above mechanism, implement PDD and NPS
			* Separate the handling from the server implementation - as little hardwiring as possible
			* Implement using a different server implementation - a spring boot one perhaps - wrapping jetty or tomcat

			The above will probably take me at least a week to nail it. But we should be in a pretty good place by the end of it - with a test server more or less there, and we can start expanding it to cover all the test cases we need.

Stuart: /* Day 4: 09/09/2021 */

2021-09-09T13:49:04Z

Day 4: 09/09/2021

← Older revision		Revision as of 13:49, 9 September 2021
Line 57:		Line 57:

	=== Day 4: 09/09/2021 ===		=== Day 4: 09/09/2021 ===
			* So, I got all of this up and running on Linux too - actually makes the terminal and python run much smoother - it's kinda built for that sort of environment.
			* I did an analysis of the two server files and deleted (pretty much) everything which they don't use. Quite a bit easier to see what's there now.
			* I also analysed NPS as much as I could without a valid config to use or example client calls. This is available at [[Noctam/DWP/ACS_Test_Harness/Notes#NPS_Handling\|Notes/NPS Handling]]

			==== The rest of the day ====
			I want to start designing really, so I'm going to spend some time with pen and paper to see what I can come up with in the next couple of hours.

	=== Day 5: 10/09/2021 ===		=== Day 5: 10/09/2021 ===

Stuart: Moved dates into Day headers; added placeholders for future dates

2021-09-09T12:31:10Z

Moved dates into Day <n> headers; added placeholders for future dates

← Older revision		Revision as of 12:31, 9 September 2021
Line 1:		Line 1:
	<noinclude>{{Nav/ACS Test Harness}}</noinclude>		<noinclude>{{Nav/ACS Test Harness}}</noinclude>
	== Day 1 ==		=== Day 1: 06/06/2021 ===
	06/06/2021

	Initial workshop to discuss requirements and broad development strategy.		Initial workshop to discuss requirements and broad development strategy.
	See [[Noctam/DWP/ACS Test Harness/Meetings/01-Initial Workshop\|Meetings: Initial Workshop]].		See [[Noctam/DWP/ACS Test Harness/Meetings/01-Initial Workshop\|Meetings: Initial Workshop]].

	== Day 2 ==		=== Day 2: 07/09/2021 ===
	07/09/2021

	In which I:		In which I:
	* installed and set up intellij + python on my noctam PC		* installed and set up intellij + python on my noctam PC
Line 35:		Line 31:
	{{quote\|OK, this is actually quite flexible, and, although the code is terse and uncommented, it's not that hard to follow. Ultimately, it doesn't do that much (in the above, quite narrow, use case) - I could fairly easily strip out everything that isn't required by a DCI call and just have that as our starting block of code... I might do that, though I want to get PDD and NPS in there too, neither of which are handled in my initial examination above. }}		{{quote\|OK, this is actually quite flexible, and, although the code is terse and uncommented, it's not that hard to follow. Ultimately, it doesn't do that much (in the above, quite narrow, use case) - I could fairly easily strip out everything that isn't required by a DCI call and just have that as our starting block of code... I might do that, though I want to get PDD and NPS in there too, neither of which are handled in my initial examination above. }}

	== Day 3 ==		=== Day 3: 08/09/2021 ===
	08/09/2021

	=== Plan ===		==== Plan ====
	* Step through PDD processing in the CISSIM program		* Step through PDD processing in the CISSIM program
	* Step through NPS processing too		* Step through NPS processing too
Line 46:		Line 41:
	** I'll probably need to talk through some aspects of it (inputs / outputs) with the FDACS team, maybe tomorrow for that.		** I'll probably need to talk through some aspects of it (inputs / outputs) with the FDACS team, maybe tomorrow for that.

	=== Execution ===		==== Execution ====
	* Well, I stepped through the PDD processing		* Well, I stepped through the PDD processing
	** You can see the results at [[Noctam/DWP/ACS Test Harness/Notes#PDD Handling\|Notes/PDD Handling]].		** You can see the results at [[Noctam/DWP/ACS Test Harness/Notes#PDD Handling\|Notes/PDD Handling]].
Line 54:		Line 49:
	* I decided to take a half day, because I'm really feeling like I need the real stuff rather than the piecemeal bits that I'm looking at here. Gives me chance to catch up on some home chores too.		* I decided to take a half day, because I'm really feeling like I need the real stuff rather than the piecemeal bits that I'm looking at here. Gives me chance to catch up on some home chores too.

	=== Plans for tomorrow ===		==== Plans for tomorrow ====
	* Look through NPS and document it properly.		* Look through NPS and document it properly.
	* Modify the scripts to start and stop the NPS Sim server too		* Modify the scripts to start and stop the NPS Sim server too
Line 60:		Line 55:
	* Start stripping away the stuff that we don't immediately need for ACS Test Harness		* Start stripping away the stuff that we don't immediately need for ACS Test Harness
	* Broad design for the java CIS Sim server, with respect to ACS dev testing		* Broad design for the java CIS Sim server, with respect to ACS dev testing

			=== Day 4: 09/09/2021 ===

			=== Day 5: 10/09/2021 ===

Stuart: /* Execution */

2021-09-08T12:44:11Z

Execution

@@ Line 50: / Line 50: @@
 ** You can see the results at [[Noctam/DWP/ACS Test Harness/Notes#PDD Handling|Notes/PDD Handling]].
 * I identified a few things which will need looking into - see [[Noctam/DWP/ACS Test Harness/Notes#CISSIM|Notes/CISSIM]] - the config processing is probably the main one since it's needed for PDD and it does some pretty meaty stuff at config loading time to determine the locations for the fields within PDD data blocks.
-* NPS is at a bit of an impasse - I might shelve it to look at the config a bit more.
+* I picked up some new setup instructions, which included reference to scripts that Sean has written to start and stop the servers - plural, i.e. the NPS server as well. Duh.

Stuart: /* Day 3 */ Late morning update

2021-09-08T10:00:13Z

Day 3: Late morning update

← Older revision		Revision as of 10:00, 8 September 2021
Line 47:		Line 47:

	=== Execution ===		=== Execution ===
			* Well, I stepped through the PDD processing
			** You can see the results at [[Noctam/DWP/ACS Test Harness/Notes#PDD Handling\|Notes/PDD Handling]].
			* I identified a few things which will need looking into - see [[Noctam/DWP/ACS Test Harness/Notes#CISSIM\|Notes/CISSIM]] - the config processing is probably the main one since it's needed for PDD and it does some pretty meaty stuff at config loading time to determine the locations for the fields within PDD data blocks.
			* NPS is at a bit of an impasse - I might shelve it to look at the config a bit more.

Stuart: Day 3 stub

2021-09-08T07:21:52Z

Day 3 stub

← Older revision		Revision as of 07:21, 8 September 2021
Line 34:		Line 34:
	* Stepped it through a DCI913 call to get an idea of a) what it did, and b) how it did it. My [[Noctam/DWP/ACS Test Harness/Notes#CISSIM\|notes]] are available, for now, but in a nutshell:		* Stepped it through a DCI913 call to get an idea of a) what it did, and b) how it did it. My [[Noctam/DWP/ACS Test Harness/Notes#CISSIM\|notes]] are available, for now, but in a nutshell:
	{{quote\|OK, this is actually quite flexible, and, although the code is terse and uncommented, it's not that hard to follow. Ultimately, it doesn't do that much (in the above, quite narrow, use case) - I could fairly easily strip out everything that isn't required by a DCI call and just have that as our starting block of code... I might do that, though I want to get PDD and NPS in there too, neither of which are handled in my initial examination above. }}		{{quote\|OK, this is actually quite flexible, and, although the code is terse and uncommented, it's not that hard to follow. Ultimately, it doesn't do that much (in the above, quite narrow, use case) - I could fairly easily strip out everything that isn't required by a DCI call and just have that as our starting block of code... I might do that, though I want to get PDD and NPS in there too, neither of which are handled in my initial examination above. }}

			== Day 3 ==
			08/09/2021

			=== Plan ===
			* Step through PDD processing in the CISSIM program
			* Step through NPS processing too
			* Document the aboves
			* Start stripping out unused files / classes and reduce the program to its minimum codeset for ACS testing
			* Start a design of the java version of it
			** I'll probably need to talk through some aspects of it (inputs / outputs) with the FDACS team, maybe tomorrow for that.

			=== Execution ===

Stuart: /* Day 2 */

2021-09-07T16:45:21Z

Day 2

← Older revision		Revision as of 16:45, 7 September 2021
Line 32:		Line 32:
	checks of the config value starts with '../' and replaces '../' with the		checks of the config value starts with '../' and replaces '../' with the
	root folder provided by getrootfolder()) when it loads the config.		root folder provided by getrootfolder()) when it loads the config.
			* Stepped it through a DCI913 call to get an idea of a) what it did, and b) how it did it. My [[Noctam/DWP/ACS Test Harness/Notes#CISSIM\|notes]] are available, for now, but in a nutshell:
			{{quote\|OK, this is actually quite flexible, and, although the code is terse and uncommented, it's not that hard to follow. Ultimately, it doesn't do that much (in the above, quite narrow, use case) - I could fairly easily strip out everything that isn't required by a DCI call and just have that as our starting block of code... I might do that, though I want to get PDD and NPS in there too, neither of which are handled in my initial examination above. }}

Stuart: First cut of first week

2021-09-07T15:39:32Z

First cut of first week

New page

<noinclude>{{Nav/ACS Test Harness}}</noinclude>
== Day 1 ==
06/06/2021

Initial workshop to discuss requirements and broad development strategy.
See [[Noctam/DWP/ACS Test Harness/Meetings/01-Initial Workshop|Meetings: Initial Workshop]].

== Day 2 ==
07/09/2021

In which I:
* installed and set up intellij + python on my noctam PC
* created simplistic {{c|sim-start}} and {{c|sim-stop}} scripts to start and stop the CISSIM server
* got it up and running, which required a bit of jiggery-pokery:
** needed to install {{c|windows-curses}} in my python environment: {{c|pip install windows-curses}}
** Fixed a couple of relative paths hardcoded into a couple of python files. To wit:

commit 5205c969edcfe9e075ca8261a3ca2f13fa3e7129
Author: Stuart Wood (Noctam) <stuart.wood@noctam.co.uk>
Date: Tue Sep 7 14:34:42 2021 +0100

Don't rely on relative paths for files/folders

Adds a 'getrootfolder' function to pdutils which gets the root folder of
the 'project', by getting the parent of the directory in which the
pdutils file is found.

I can't immediately think of any problem with this, and it means you can
run it safely from anywhere.

This also dynamically alters the 'FOLDERS' config value (actually it
checks of the config value starts with '../' and replaces '../' with the
root folder provided by getrootfolder()) when it loads the config.