Noctam/DWP/ACS Test Harness/Existing System/CISSIM/Command Line Tools
ACS Test Harness
|
|---|
|
Meetings Existing System Specifications Diary |
There are many command line tools to do various things within the CISSIM / proxy / data management / testing stub / kitchen sink application.
This page tries to exhaustively go through all of them. It fails. But it continues to try.
File formats
There are a few file formats used here.
base64-ebcdic
EBCDIC data and binary fields encoded into base64.
This is the form for most PD calls - the content is a record whose definition is held elsewhere (in COBOL copybook-ish form)
Used as input and output to/from pdutils and pdedit.
base64-ascii
ASCII data and binary fields encoded into base64.
Just like base64-ebcdic except the text fields are encoded into ASCII rather than EBCDIC.
This can be used as input to pdutils when using the CONVERT2E option to convert it to base64-ebcdic, and output from pdutils using the CONVERT2A option.
Filename format
Some of the following utilities expect the filename to be in one of a number of specific formats. They use this format to embed information about the transaction that the file represents, later extracted by the programs to look up related data and/or structures.
pdconvert formats
The pdconvert, pdconvertone and pdconvertall programs expect the files to be named in one of two dot-separated formats; one with 4 'parts', one with any other number of 'parts' as long they number at least 3. The 4 part format is represented by the regex:
^(?P<dialognum>[^.]{0,6})\.[^.]*\.[^.]*\.(?P<direction>in|out)$
eg. "DCI908.NINO.STUB.out"
The 4 dot-separated parts indicate that this is an 'R' (read) mode transaction. A suffix of 'in' indicates a 'RQST' direction, 'out' indicates a 'RPLY' direction. The dialog is extracted as the first 6 chars of the first part, the mode and the direction separated by underscores, so the above example would yield:
DCI908_R_RQST
The 'any number above 2, except 4' format (?) extracts more information from the filename:
^(?P<dialognum>[^.]*)\.[^.]{3}(?P<messageid>[^.]{4})(?P<txnmode>[^.]).*\.(?P<direction>in|out)$
eg. "PD375.CIS6084RCURRINT.153250.634934c5c8d6.in"
This builds the dialog and transaction mode from the filename. The direction is mapped to 'RQST' or 'RPLY' where the filename
has 'in' or 'out', and the dialog is built up as: <dialognum>_<messageid>_<txnmode>_<direction>. The above example
would end up with the dialog:
PD375_6084_R_RQST
Programs
pdutils
Primarily a library file, used throughout various calls from the daemon programs as well as other utilities. It can, however, also be called directly from the command line.
Usage: pdutils <DUMP/OUT/SAVE/CONVERT2E/CONVERT2A> <dialog> <RQST/RPLY> <requestfile/replyfile>
Note that SAVE requires redirected input from an ASCII text string, such as '< <datafile>'
and that CONVERT requires redirected input from B64 encoded ASCII data '< <datafile>'
DUMP
Dumps the structure of a request found in a base64-encoded EBCDIC file out to the console.
This puts everything out - ie. all the fields, the fillers and the groups. The values are wrapped into square brackets the appropriate size for capturing the data for the field. Binary values have leading zeroes trimmed; all char values are displayed as found.
It prefixes the field with the data structure (eg B4 for binary 4 bytes; X16 for char 16 bytes; N2 for numeric char 2 bytes).
It suffixes the value with the hex value found for the field, separated into space-separated octets, wrapped in parantheses.
OUT
Dumps the structure of a request found in a base64-encoded EBCDIC file out to the console.
This only puts out the fields and fillers. The values are wrapped into square brackets the appropriate size for capturing the data for the field. Binary values have leading zeroes trimmed; all char values are displayed as found.
SAVE
Note: this doesn't work on Windows, because it uses select.select() to ensure that stdin is available, and that function only works on file descriptors on POSIX systems.
Uses a 'hidden' fifth argument, i.e. it expects the input file to be provided on stdin. The <requestfile/replyfile> argument is, in this case, an output file.
I have tweaked the code to work in Windows and... I don't really know what this does, i.e. I don't know what format the input file is expected to be in. It's not the same as the 'requestfile/replyfile' used for DUMP and OUT, because that fails with the exception message: invalid literal for int() with base 10: 'AAAAMMPJ4'
CONVERT2E
Note: this doesn't work on Windows, because it uses select.select() to ensure that stdin is available, and that function only works on file descriptors on POSIX systems.
Uses a 'hidden' fifth argument, i.e. it expects the input file to be provided on stdin. The <requestfile/replyfile> argument is, in this case, an output file.
This converts input from stdin, in base64-encoded ASCII record format to the output file, in base64-encoded EBCDIC record format. This is the exact mirror of CONVERT2A.
Note: This only works for PD data, not DCI or NPS (which are only ever required in ASCII-based encodings).
CONVERT2A
Note: this doesn't work on Windows, because it uses select.select() to ensure that stdin is available, and that function only works on file descriptors on POSIX systems.
Uses a 'hidden' fifth argument, i.e. it expects the input file to be provided on stdin. The <requestfile/replyfile> argument is, in this case, an output file.
This converts input from stdin, in base64-encoded EBCDIC record format to the output file, in base64-encoded ASCII record format. This is the exact mirror of CONVERT2A.
Note: This only works for PD data, not DCI or NPS (which are only ever required in ASCII-based encodings).
pdedit
Edits the PD and DCI call data via a curses interface, using the COBOL data layouts for the calls.
Usage: pdedit <dialog> <RQST/RPLY> <sourcefile> <destfile>
The 'sourcefile' in this case is the same as the 'requestfile/replyfile' provided in pdutils, that is, a file containing base64-encoded EBCDIC data. The 'destfile' is in the same format with the changes applied (assuming that you save inside pdedit).
Intriguingly, it appears to ignore the 'RQST/RPLY' in the dialog and effectively overwrites it with the RQST/RPLY argument given to the program. Not really sure why... that dialog is a bit of a hack, really, I suspect just to avoid multiple keys or too much structure in the config.
PD350/1151/RQST/R
might have worked better, with a structured system, be it XML or JSON or YAML or whatever. Since I don't really understand which things would be better grouped together, I'll leave this as an idle thought for now.
pddb
Creates a database entry which effectively maps a response file onto a request.
Usage: pddb <dialog> <sourcefile> <destfile> <EBCDIC/ASCII>
The output is in the form:
<dialog>:<response-file>:<input-field-name=value[;...]>
So, for a PD350 1151 request, you might see something like:
$ python cissim/pddb.py PD350_1151_R_RQST requests/pd350-request-e messages/PD350.CIS1151RSTRTDIA.AASTUB.out EBCDIC PD350_1151_R_RQST:PD350.CIS1151RSTRTDIA.AASTUB.out:H1151-NINO=ST000001;H1151-NINO-SFX-TX=A;H1151-RQST-DSS-DAYS=20190213;H1151-RQST-DIA-ID=PD350 ;H1151-RQST-USR-NS-LVL=0000;H1151-RQST-USR-LS-LVL=0000;H1151-RQST-USR-BUS-SYS-NO=43;H1151-DEPT-ID-TP=1;H1151-OFF-LOC-NO=04710;H1151-RQST-SPCFC-BUS-TP=85;
These entries can be appended into a single file, with each entry on a separate line and work together to a form a readonly database of sorts, which the server can use to lookup the response to use based on the values of the inputs provided in the request.
pdserve
This is what uses the given 'database' files from pddb to serve the data based on the requests.
Usage: pdserve <configfile> [<port>]
It takes a config file which is effectively made up of paths to the 'databases' built up using pddb along with an (optional) SSL flag. Each pddb line in the config is something like:
<path-to-db-file>:<folder>
So, for example:
configfile.ja030
SSL=N pdbase/JA030_01.CIS.txt:pdbase pdbase/JA030_02.CIS.txt:pdbase pdbase/JA030_99.CIS.txt:pdbase
pdserve will expect to find the response files with the names defined in the 'database' within the folder specified in the config, so you could have multiple entries here pointing to different databases.
The 'port' here is the port it will listen on. This is analogous to a 'swimlane' (<sigh>) and, indeed, the terms are used interchangeably in some places. The default port value used is 9127.
The recommended way of running this is using nohup so that you can exit the shell and it will continue to run... something like:
nohup python pdserve.py configfile.ja030 9130 >ja030.log &
... and of course, the only way to kill it after doing that is by identifying the process that's running and using the kill command.
Final thing. Quite important.
The version I have doesn't work.
In Python 2:
$ python2 ../cissim/pdserve.py pdserve.test.config 9927
File "../cissim/pdserve.py", line 68
print(s.server_version, end=' ')
^
SyntaxError: invalid syntax
In Python 3:
$ python ../cissim/pdserve.py pdserve.test.config 9927
Traceback (most recent call last):
File "../cissim/pdserve.py", line 316, in <module>
status = main()
File "../cissim/pdserve.py", line 260, in main
if '#' in line:
TypeError: a bytes-like object is required, not 'str'
It looks like it's been half-heartedly converted from 2 to 3, but untested. I'm assuming that either nobody uses it, or it's been fixed in place wherever it is used and the changes haven't been reflected back to the source.
pdcall
(Incidentally, this was a bit messy to get working due to using pycurl, I had to apt install libcurl4-openssl-dev libssl-dev and pip install pycurl in order to get this to work on my Ubuntu machine - no idea what this will require on Mac, but be prepared for pain, given how little access we have.)
This can be used to call the server and test that it's working correctly.
Usage: pdcall <swimlane> <dialog> <requestfile> <replyfile>
The '<swimlane>' here is the port that the server is listening on.
This opens the request file and displays it using pdedit. After editing and saving, it posts the updated request to the server, captures the response and then opens that in pdedit too so that you can see (and, indeed, edit) the response, before saving to the output file.
At least, I think that's what it does - because I couldn't get pdserve running, I had very little to test, so this is conjecture and glancery.
I've just dumped all the 'pd' python files here for further processing.
pdacs
Calls a running ACS instance with an edited request, storing the response into the required output file.
Usage: pdacs <service> <dialog> <requestfile> <replyfile>
This pops up a pdedit window to edit the request (actually saving the resultant file to the given location), then creates the
request by prepending the ACS input header, with the uppercased service embedded in it, to the input file contents.
The ACS input header is fixed (service aside) and has the following content:
0ed1249b558f||True|101307| |DCP26PD350 |null|<service>|11606730|null|DWDDE662|2|1548331293|0|500|0|false|P|60|468|
The full input is saved to a file in the current directory called <requestfile>.acs. The response from ACS is saved into the
provided replyfile.
pdconvertall
Converts *.in files and corresponding *.out files from a source folder into a destination folder.
Usage: pdconvertall.py <sourcefolder> <targetfolder>
Note that the filename format is significant for this program and, natch, undocumented. See pdconvert formats for more details.
This scans the sourcefolder for all files matching *.in which do not start with the characters NPS and
have corresponding *.out files and 'converts' them to a text-based format for use with pdserved.
The conversion of the input file is almost identical to the processing done by calling pdutils OUT with the dialog and direction specified explicitly, though this program picks up that information from the embedded data in the filename.
The directory structure it creates is:
<outputfolder>
|- <dialog>
|- <rundate-yyyyMMdd>
|- <nino>.DATA
The format is text-based in the style of the pdutils OUT format. The pdserved server then reads from the directory when it is processing the requests and determining the responses to return. The files can be modified in place, or new files can be placed there. The server will pick up any external changes made to these files.
pdconvertone
This is a thin wrapper around pdconvertall which works for a single input and output file. That is, it
still expects a *.in and corresponding *.out file to convert, and it emits the output into the directory
structure described in pdconvertall.
Usage: pdconvertone.py <sourcespec> <targetfolder>
This calls pdconvertall with the single file identified by sourcespec. Note that if the file begins with the
characters NPS or there is no corresponding *.out file, it will be skipped
pdconvert
Converts a single file into its text-based format, emitting the output to stdout.
Usage: pdconvert.py <pdfile>
This does not have the same restrictions as pdconvertall and pdconvertone - i.e.
it does not require corresponding *.in and *.out files, though the given file's filename must be one of those.
It will still not work for NPS files (though, unlike the above programs, it will try and probably crash and burn in the
attempt).
pddata
Library file.
This has a load of functions in it, only some of which I understand.
The main one appears to be getninodata which, typical cissim style, takes 10 arguments (interrobang?!) and has...
well, some documentation which concentrates on one or two arguments and expects a fairly solid understanding of
what it's doing before you start reading.
I'm going to have to come back to this to try and grasp it.
Note: This technically can be executed, but it looks like the stuff in the main call is just development stuff - testing other functions and such like.
pddump
This dumps the data from the given (base64-encoded EBCDIC) file to the console in text form with square brackets to enclose the data values. Note: This uses the pdconvert format for the filename.
Usage: pddump.py <pdfile>
This is a call to pdutils DUMP which takes the dialog / direction arguments from the filename format rather than requiring them to be stated explicitly.
pdout
This dumps the data from the given (base64-encoded EBCDIC) file to the console in text form with square brackets to enclose the data values. Note: This uses the pdconvert format for the filename.
Usage: pdout.py <pdfile>
This is a call to pdutils OUT which takes the dialog / direction arguments from the filename
pdparams
Library file.
Contains a single function: processparams. This function also exists in pdutils, and it looks like that
one is used but this one is not. I presume it's obsolete, though there's no indication of that in the code.
Deprecated?
pdproxy
A very straightforward proxy which writes the request and response of any CIS calls made to it to files in the current directory.
Usage: pdproxy <swimlane> <configfile> <folder> [<port>]
The swimlane determines whether the 'real' CIS is called or whether the 'SIM' is called. Basically, if the swimlane
argument is in the ranges: 14-15 or 17-28 (both inclusive), onward calls from the proxy will go to CIS proper. Any other
value is treated as the port that the simulator is listening on and it will be connected to using that port number.
pdreadall
Reads all registered dialogs and calls them for the specified NINo and optional surname.
Usage: pdreadall <swimlane> <paramfile> <folder> <nino> [<surname>]
pdreadoneedit
Calls a specified dialog in CIS (or a CIS simulator) for a given NINo and optional surname. This allows the editing of the record to be sent in the request before the call is made. The input and output files are created (I think) in the current directory with the appropriate filename format.
Usage: pdreadoneedit <swimlane> <paramfile> <dialog> <nino> [<surname>]
pdreadone
Calls a specified dialog in CIS (or a CIS simulator) for a given NINo and optional surname. This does not allow the editing of the record to be sent in the request before the call is made. The input and output files are created (I think) in the current directory with the appropriate filename format.
Usage: pdreadone <swimlane> <paramfile> <dialog> <nino> [<surname>]
pdscreen
Library file.
This file contains a single class 'Textbox' which provides the editing widget used in pdedit to
edit a value in a record file.
pdserved
An upgraded version of pdserve with NPS support and separate message directories.
Usage: pdserved.py <port> -P <pddfolder> -D <dcifolder> [-N <npsfolder>]
Aside from working in Python3 and adding the NPS handling which was added in pdservep, this primarily implements the separation of the data directories into 'PD', 'DCI' and 'NPS' folders. These could all be the same folder (I think), but the option is there now to have different ones for each type. Presumably, you might want different PD responses for different services, but you want to use the same DCI/NPS responses each time, so this saves on a load of copying and pasting.
pdservedt
pdserved with a process_wait() function, which loads the timing data from the hardcoded location
at ../serve/transtimes.py. Not ideal - it requires running the program from a specific directory, and, of course,
this is undocumented (as far as I can see anyway).
The timing data is a map of timings against dialog; the example in the code is:
# timings are a list, first tuple is times to cycle through, rest are exceptional count/time pairs # 'CP650_1151_R_RPLY': [(0.15, 0.25, 0.21, 0.17, 0.20, 0.16), (20, 2.3), (100, 28)],
Basically, it will cycle through the first tuple of timings, except when the count for a given dialog reaches the first value in any of the following tuples. If it does (in this case, the 20th and 100th time a dialog is processed within the server), it will use the timing associated with that number. Reasonably straightforward.
Note: This seems the most sensible one to draw a 'proper' version from. If we want to re-do the timing, then we could branch from pdserved instead... it certainly needs changing a bit (the config is dire - a hardcoded filename in a directory reached using a relative path... ugh), and it's quite simplistic, but it does work.
pdservep
This is pdserve with more NPS handling (and fixed for Python3).
The NPS changes in here appear to be included in pdserved too, so this and pdserve, at first glance, would seem to be obsolete.
pdshow
Shows and edits a base64-encoded file, extracting the dialog and direction from the filename.
Usage: pdshow.py <pdfile>
This is a wrapper around a call to pdedit which extracts the dialog and direction from the filename, rather than needing it specified explicitly.
pdstub
Stub server which provides more flexible lookup and response-modification than pdserve.
Usage: pdstub <configfile> [<port>]
This program provides a basic server which serves PD, DCI and NPS. Rather than a straight NINo/Rundate lookup, it uses search parameters to determine which response file to return for a request, and it allows replacement of data in that response file with static parameters, or inputs derived from the request.
It uses a single messages folder, rather than the separated responses folder of pdserve and each request/response mapping is defined in a single line within the config file.
pdstubt
This is pdstub with a process_wait() function, which works exactly like the same function in
pdservedt.
pdstubtt
This is pdstubt with a multi-threaded HTTP server. It's a fairly trivial change to the code; I haven't looked through it to see if there are any thread localisation issues in there - I'm assuming not, because this is being used as is at the moment, and that would probably have been noticed by now.
This seems to me to be the most mature of the pdstub implementations - multithreading the server is a no-brainer, in my humble, and the timings can be ignored. It still has the problem of a hardcoded timing config, but that shouldn't be too hard to fix in a less fragile manner.
pdtux
A Tuxedo server which advertises a few tux operations, namely:
- ping
- DCI913
- DCI923
- VMERCIS
- RESETCOUNTER
This extracts the data out of the request packet and gets the data using pddata.getninodata(), returning the resultant data (the whole thing for DCI calls, separated CIS header and W90 header for PDD calls).
There are no arguments to this program, but it does require the installation of the 'tuxedo' python module.
And the non-'pd' python files:
certs
Library file
Holds the paths to the various certificates used within cissim. 2 groups of cert and cacert files for connecting to DCI and PD; 2 groups of cert and private key files for serving.
Used in pdcall, pdserve and pdstub programs, and in some test functions in sslutils.
checkport
Redundant
Checks if it's possible to connect to localhost:9127. That's it really.
Basically, a python version of:
netstat -nap | grep ':9127.*LISTEN'
dci9n3rec
Library file
Technically, this can be called from the command line, but it seems a bit pointless. It appears to read in a file called 'in' in the current directory and, uh, loads it into a DCI9n3 record structure and prints it. I guess it's for testing purposes.
Anyway, this contains the definition of a DCI913 or DCI923 record, packed to byte alignment for the fields, with the 12 fields and their sizes defined as fixed length byte arrays.
It's only used in pdtux for extracting the data from an incoming DCI struct. The equivalent for a PD record is w80construct
extractodis
Executable
Usage: extractodis.py <log_filename> [<PID>] [<starttime>] [<endtime>]
This is completely out there. So. It takes the filename and creates 3 shell scripts with names based on it, which perform some function on the payload (base64-encoded ASCII). These scripts are:
<filename>.write.sh: to write the transaction for ACS emulation (just writing the record out)<filename>.conv.sh: to write the transaction for CIS emulation (calling pddump.py CONVERT2E)<filename>.db.sh: to write the transaction to the 'database' for serving (calling pddb)
Aside from some very ropey python issues (mix of tabs and spaces in leading whitespace - very verboten in python), it's just a very odd way of doing things.
This is the oldschool format; it's dealing with the records as is, rather than moving them to the date/nino-based database used by pdserved and beyond; the pddb output is used in the original pdserve program too. It's still valid for the 'write.sh', I guess, but I suspect that's all.
fpcapprec
Library file
Obsolete
This was used in pdtux, but is no longer used. The using has been deleted, so I can't see the context.
getfieldoffset
Test file
Obsolete
I'm pretty sure this is just testing that the pdlayouts.getdialogfieldoffset() method is working correctly.
key
Library file
This provides a few constants for keycodes, used in pdedit and pdscreen.
To be honest, it really needn't be here. The function key and some other constants are already defined in the curses
module; most of the rest are defined in the curses.ascii module. UP_LIST is home-made, but really doesn't
deserve its own file; any home-grown key values could go in the pdscreen file.
loadninos
Library File and Executable
Usage: loadninos.py <office-file>
Used in pdstub and its variants.
Loads NINOs from a CSV file (actually looks like a 'Whitespace-separated variables' style file, though it has a
name of csvfile within the code). This looks for input in the form:
SVC Office JSOV 109809 JSRZ 107476 JSFP 107038 JSLH 103401 JSKR 103838 JSLF 102636 JSGR 102805 JSRT 100598
The above example is half-inched from the code. The code is ferociously simplistic, but I guess it does the job.
Anyway, it takes the 'Office' from the lines in the 'CSV' file, strips the leading digit from it, then looks for another 'csv' file with that office number as a filename, in the same directory as the provided CSV file, so the above will look for '09809.csv', '07476.csv' etc. It will load that file, taking the first 8 chars of each line as a NINo and then maps the office to that NINo, such that each NINo has the office number of the file that it was loaded from mapped against it.
This is used in pdstub to get the office for a particular NINo, which is then used as part of the
response in a DCI913 call. The stub loads the NINos using this function passing the OFFICECSVFILE config
parameter value to load them from. It doesn't handle errors well, but I guess once it's running, you don't
change it that much.
logextract
Executable
Usage: logextract.py <logfile>
Extracts a DCI913CopyRecord from a logfile. It's a bit hacky - in the middle of it, it does some strange
stuff with the string '127.0.0.1'; I presume the logfile isn't really designed for extracting records from and
so this has to hardcode some messing around into it. It's very fragile - logfiles aren't typically structured
output (or, at least, not structured in this way).
npsload
Executable - this is an executable only; no part of it is called by other programs / libraries.
Usage: npsload <folder> <csvfile> <listfile>
This loads the csvfile from folder (i.e. the file is just the filename, not the /path/to/file),
and writes all output, including the listfile in the same folder.
The CSV file is defined as:
Entry/ies Description 1 NINo 2 Surname 3 Office 4 LOP 25x Past 5-20 NUBS2 record x 4 21-36 LOP record x 4 37 Data not found flag 38 BFM Return Code 39 RDS Code
Each NUBS2 record and LOP record is made up of 4 comma-separated entries, with the fields:
Entry Description 1 Year 2 x25 3 x50 4 Priority
This program loops through all the NINo entries, and writes an XML file for each one, calling it NPSrply.<nino>.xml,
adding an entry into the listfile for each file it created. It adds a final 'NINO not found' entry to the listfile
and exits.
paths
Library file
Contains the paths used for calls to PD, DCI and NPS, as well as a few functions which operate on and around those paths.
This is used liberally throughout the rest of the CISSim suite.
ports
Library file.
This just holds all the ports for the CIS swimlanes and returns the appropriate ports for a given swimlane when asked, in the form (pdd-port, dci-port, host).
selectserver
Executable
I have no idea what this is trying to do. It's messing around with servers and message queues, but I'm a bit lost.
sslutils
Library file
Contains functions to call servers using SSL certificates. Used in pdacs, pdcall, pdproxy and teststub
temp
Obsolete
Seems to be a copy of pdserved with some testing code in there.
teststub
Obsolete
Tests connecting to the CIS Sim using curl and a socket. Prints things. Not particularly useful.
utils
Library file
Obsolete
A variety of utility functions; apparently not used anywhere.
vmeutils
Library file and Executable
Usage: vmeutils <D/B> <date>
Some VME date functions and a small utility executable to convert a date into a VME date ('B', obviously) or
a VME date into a date ('D'). A 'VME date' in this context is the number of days since 29/02/1852.
w80construct
Library file
Like dci9n3rec, this can be called from the command line, but it also seems a bit pointless. I guess it's for testing purposes.
Anyway, this contains the definition of a W80 record, packed to byte alignment for the fields, with the various fields and their sizes defined mostly as fixed length byte arrays, though there are a few integer and long integer fields defined too - these will be compiler-dependent, so I guess that's how the tux that it's interfacing with/mocking works too?
It's only used in pdtux for extracting the data from an incoming DCI struct. The equivalent for a DCI record is dci9n3rec
Files in pdlayouts
pdlayouts/PDM
The 'PD Map' contains the fields used in the CIS header and the W80 PD header, including their name, data type and length and
an optional 'style' type argument which provides metadata for the field, eg. nino8 indicates the first 8 chars of
a NINo, runtime8 indicates that it contains a time in the format %H%M%S%f, vmedate indicates that the field
contains a 'VME date', ie. the number of days since 29/02/1852.
These 'style' arguments are used in pdutils.makerequestfile to autopopulate various fields for input based on the static
'RUNDATE' parameter value.