Noctam/DWP/ACS Test Harness/Existing System/CISSIM/Command Line Tools

ACS Test Harness

edit this panel Meetings 06/09/2021: Initial Workshop 01/10/2021: Current System Existing System CISSIM Command Line Tools User Docs PDSIM.odt PDSIM demo.odt PVTstub.odt simcertshowto.txt SIMD.odt Specifications JIRA Issues ACS Test Harness General Services CIS NPS Transactions DCI908 DCI913 DCI923 PDD Diary 2009: Week 1: 06/09 - 12/09 Week 2: 13/09 - 19/09 Week 3: 20/09 - 26/09 Week 4: 27/09 - 03/10 Notes

There are many command line tools to do various things within the CISSIM / proxy / data management / testing stub / kitchen sink application.

This page tries to exhaustively go through all of them. It fails. But it continues to try.

File formats

There are a few file formats used here.

base64-ebcdic

EBCDIC data and binary fields encoded into base64.

This is the form for most PD calls - the content is a record whose definition is held elsewhere (in COBOL copybook-ish form)

Used as input and output to/from pdutils and pdedit.

base64-ascii

ASCII data and binary fields encoded into base64.

Just like base64-ebcdic except the text fields are encoded into ASCII rather than EBCDIC.

This can be used as input to pdutils when using the CONVERT2E option to convert it to base64-ebcdic, and output from pdutils using the CONVERT2A option.

pdutils

Primarily a library file, used throughout various calls from the daemon programs as well as other utilities. It can, however, also be called directly from the command line.

Usage: pdutils <DUMP/OUT/SAVE/CONVERT2E/CONVERT2A> <dialog> <RQST/RPLY> <requestfile/replyfile>

  Note that SAVE requires redirected input from an ASCII text string, such as '< <datafile>'
       and that CONVERT requires redirected input from B64 encoded ASCII data '< <datafile>'

DUMP

Dumps the structure of a request found in a base64-encoded EBCDIC file out to the console.

This puts everything out - ie. all the fields, the fillers and the groups. The values are wrapped into square brackets the appropriate size for capturing the data for the field. Binary values have leading zeroes trimmed; all char values are displayed as found.

It prefixes the field with the data structure (eg B4 for binary 4 bytes; X16 for char 16 bytes; N2 for numeric char 2 bytes).

It suffixes the value with the hex value found for the field, separated into space-separated octets, wrapped in parantheses.

OUT

Dumps the structure of a request found in a base64-encoded EBCDIC file out to the console.

This only puts out the fields and fillers. The values are wrapped into square brackets the appropriate size for capturing the data for the field. Binary values have leading zeroes trimmed; all char values are displayed as found.

SAVE

Note: this doesn't work on Windows, because it uses select.select() to ensure that stdin is available, and that function only works on file descriptors on POSIX systems.

Uses a 'hidden' fifth argument, i.e. it expects the input file to be provided on stdin. The <requestfile/replyfile> argument is, in this case, an output file.

I have tweaked the code to work in Windows and... I don't really know what this does, i.e. I don't know what format the input file is expected to be in. It's not the same as the 'requestfile/replyfile' used for DUMP and OUT, because that fails with the exception message: invalid literal for int() with base 10: 'AAAAMMPJ4'

CONVERT2E

Note: this doesn't work on Windows, because it uses select.select() to ensure that stdin is available, and that function only works on file descriptors on POSIX systems.

Uses a 'hidden' fifth argument, i.e. it expects the input file to be provided on stdin. The <requestfile/replyfile> argument is, in this case, an output file.

This converts input from stdin, in base64-encoded ASCII record format to the output file, in base64-encoded EBCDIC record format. This is the exact mirror of CONVERT2A.

Note: This only works for PD data, not DCI or NPS (which are only ever required in ASCII-based encodings).

CONVERT2A

Note: this doesn't work on Windows, because it uses select.select() to ensure that stdin is available, and that function only works on file descriptors on POSIX systems.

Uses a 'hidden' fifth argument, i.e. it expects the input file to be provided on stdin. The <requestfile/replyfile> argument is, in this case, an output file.

This converts input from stdin, in base64-encoded EBCDIC record format to the output file, in base64-encoded ASCII record format. This is the exact mirror of CONVERT2A.

Note: This only works for PD data, not DCI or NPS (which are only ever required in ASCII-based encodings).

pdedit

Edits the PD and DCI call data via a curses interface, using the COBOL data layouts for the calls.

Usage: pdedit <dialog> <RQST/RPLY> <sourcefile> <destfile>

The 'sourcefile' in this case is the same as the 'requestfile/replyfile' provided in pdutils, that is, a file containing base64-encoded EBCDIC data. The 'destfile' is in the same format with the changes applied (assuming that you save inside pdedit).

Intriguingly, it appears to ignore the 'RQST/RPLY' in the dialog and effectively overwrites it with the RQST/RPLY argument given to the program. Not really sure why... that dialog is a bit of a hack, really, I suspect just to avoid multiple keys or too much structure in the config.

PD350/1151/RQST/R

might have worked better, with a structured system, be it XML or JSON or YAML or whatever. Since I don't really understand which things would be better grouped together, I'll leave this as an idle thought for now.

pddb

Creates a database entry which effectively maps a response file onto a request.

Usage: pddb <dialog> <sourcefile> <destfile> <EBCDIC/ASCII>

The output is in the form:

<dialog>:<response-file>:<input-field-name=value[;...]>

So, for a PD350 1151 request, you might see something like:

$ python cissim/pddb.py PD350_1151_R_RQST requests/pd350-request-e messages/PD350.CIS1151RSTRTDIA.AASTUB.out EBCDIC
PD350_1151_R_RQST:PD350.CIS1151RSTRTDIA.AASTUB.out:H1151-NINO=ST000001;H1151-NINO-SFX-TX=A;H1151-RQST-DSS-DAYS=20190213;H1151-RQST-DIA-ID=PD350 ;H1151-RQST-USR-NS-LVL=0000;H1151-RQST-USR-LS-LVL=0000;H1151-RQST-USR-BUS-SYS-NO=43;H1151-DEPT-ID-TP=1;H1151-OFF-LOC-NO=04710;H1151-RQST-SPCFC-BUS-TP=85;

These entries can be appended into a single file, with each entry on a separate line and work together to a form a readonly database of sorts, which the server can use to lookup the response to use based on the values of the inputs provided in the request.

pdserve

This is what uses the given 'database' files from pddb to serve the data based on the requests.

Usage: pdserve <configfile> [<port>]

It takes a config file which is effectively made up of paths to the 'databases' built up using pddb along with an (optional) SSL flag. Each pddb line in the config is something like:

<path-to-db-file>:<folder>

So, for example:

configfile.ja030

SSL=N
pdbase/JA030_01.CIS.txt:pdbase
pdbase/JA030_02.CIS.txt:pdbase
pdbase/JA030_99.CIS.txt:pdbase

pdserve will expect to find the response files with the names defined in the 'database' within the folder specified in the config, so you could have multiple entries here pointing to different databases.

The 'port' here is the port it will listen on. This is analogous to a 'swimlane' (<sigh>) and, indeed, the terms are used interchangeably in some places. The default port value used is 9127.

The recommended way of running this is using nohup so that you can exit the shell and it will continue to run... something like:

nohup python pdserve.py configfile.ja030 9130 >ja030.log &

... and of course, the only way to kill it after doing that is by identifying the process that's running and using the kill command.

Final thing. Quite important.
The version I have doesn't work.

In Python 2:

 $ python2 ../cissim/pdserve.py pdserve.test.config 9927
   File "../cissim/pdserve.py", line 68
     print(s.server_version, end=' ')
                                ^
 SyntaxError: invalid syntax

In Python 3:

 $ python ../cissim/pdserve.py pdserve.test.config 9927
 Traceback (most recent call last):
   File "../cissim/pdserve.py", line 316, in <module>
     status = main()
   File "../cissim/pdserve.py", line 260, in main
     if '#' in line:
 TypeError: a bytes-like object is required, not 'str'

It looks like it's been half-heartedly converted from 2 to 3, but untested. I'm assuming that either nobody uses it, or it's been fixed in place wherever it is used and the changes haven't been reflected back to the source.

pdcall

(Incidentally, this was a bit messy to get working due to using pycurl, I had to apt install libcurl4-openssl-dev libssl-dev and pip install pycurl in order to get this to work on my Ubuntu machine - no idea what this will require on Mac, but be prepared for pain, given how little access we have.)

This can be used to call the server and test that it's working correctly.

Usage: pdcall <swimlane> <dialog> <requestfile> <replyfile>

The '<swimlane>' here is the port that the server is listening on.

This opens the request file and displays it using pdedit. After editing and saving, it posts the updated request to the server, captures the response and then opens that in pdedit too so that you can see (and, indeed, edit) the response, before saving to the output file.

At least, I think that's what it does - because I couldn't get pdserve running, I had very little to test, so this is conjecture and glancery.

---

I've just dumped all the 'pd' python files here for further processing.

pdtux

---

And the non-'pd' python files: