Project 3

Due: July 19, before midnight

Important Reminder: As per the course Academic Honesty Statement, cheating of any kind will minimally result in your letter grade for the entire course being reduced by one level.

This document first describes the aims of this project followed by a brief overview. It then lists the requirements as explicitly as possible. It describes the files with which you have been provided. Finally, it provides some hints as to how the project requirements can be met.

Aims

The aims of this project are as follows:

To expose you to expressjs.
To make you design and implement REST web services.
To familiarize you with testing web services using a framework like supertest.

Requirements

You must push a submit/prj3-sol directory to your github repository such that typing npm ci within that directory followed by tsc is sufficient to run the project using ./index.mjs with usage:

$ ./index.mjs CONFIG_MJS [SS_JSON_PATH...]

will start a web server using the parameters defined in the *.mjs configuration file at path CONFIG_MJS. If the optional SS_JSON_PATH... parameters are specified, then the database is loaded with the grades contained in the JSON files at the paths SS_JSON_PATH. Note that for each JSON file, the spreadsheet name SS_NAME is derived from the basename SS_NAME.json of SS_JSON_PATH.

You are being provided with code which provides the necessary command-line behavior.

All request and response bodies for the web services must use type JSON. Responses are always enclosed within success or error envelopes as per the following typescript definitions (in response-envelope.ts).

/** a link contains a href URL and HTTP method */
type HrefMethod = {
  href: string,
  method: string
};

/** a self link using rel self */
export type SelfLink = {
  self: HrefMethod,
};

/** a result of type T which has a links containing a self-link */
type LinkedResult<T> = {
  links: SelfLink,
  result: T,
};

/** a response envelope always has an isOk and HTTP status */
type Envelope = {
  isOk: boolean,
  status: number,
};

/** an envelope for a successful response */
export type SuccessEnvelope<T> = Envelope & LinkedResult<T> & {
  isOk: true,
};

/** an envelope for a failed response */
export type ErrorEnvelope = Envelope & {
  isOk: false,
  errors: { message: string, options?: { [key:string]: string } }[],
};

Assuming that your server is set up to run with all URLs rooted at BASE, it will need to implement the following web services:

Get-Cell: GET BASE/SS_NAME/CELL_ID: This service should set the result of the success envelope to a { expr: string, value: number } representing the state of cell CELL_ID in spreadsheet SS_NAME.
Set-Cell: PATCH BASE/SS_NAME/CELL_ID?expr=EXPR: This service should update the expr for cell CELL_ID in spreadsheet SS_NAME to EXPR and set the result of the success envelope to an object { [cellId: string]: number } of all affected cells.
Copy-Cell: PATCH BASE/SS_NAME/CELL_ID?srcCellId=SRC_CELL_ID: This service should copy the expr from cell SRC_CELL_ID to cell CELL_ID in spreadsheet SS_NAME to EXPR and set the result of the success envelope to an object { [cellId: string]: number } of all affected cells.
Delete-Cell: DELETE BASE/SS_NAME/CELL_ID: This service should delete cell CELL_ID in spreadsheet SS_NAME and set the result of the success envelope to an object { [cellId: string]: number } of all affected cells.
Clear-Spreadsheet: DELETE BASE/SS_NAME: Clear the contents of spreadsheet SS_NAME. The result in the success envelope should be undefined.
Load-Spreadsheet: PUT BASE/SS_NAME: Set the contents of spreadsheet SS_NAME to the JSON body of the request which should be an array of [ string, string ] pairs representing the [ CELL_ID, EXPR ]. The result in the success envelope should be undefined.
Get-Spreadsheet: GET BASE/SS_NAME: This service should set the result of the success envelope to an array of [ string, string ] pairs representing the [ CELL_ID, EXPR ] of all non-deleted cells in spreadsheet SS_NAME.

Note that both the Set-Cell and Copy-Cell use a PATCH method on the same URL; they differ only in their query parameters. The former uses an expr query parameter, whereas the latter uses a srcCellId query parameter.

All success envelopes should have an HTTP status code 200 OK. If any of the above services encounters an error, then the service should return a suitable error envelope. Error responses should contain a suitable HTTP status (often BAD REQUEST 400).

The behavior of the program is illustrated in this annotated log.

A working version of these web services can be accessed at <https://zdu.binghamton.edu:2345> with BASE set to /api.

It is preloaded with spreadsheets for each student registered for this course with spreadsheet name set to your BU email ID (the portion of your BU email address before the @). The initial contents of these spreadsheets are identical to test1-ss.json data.
If you decide to play with this server, please use only your spreadsheet.
This URL is accessible only from within the CS network; you should be able to access it from your VM.
For example, you can list out your spreadsheet's data by using:
```
    $ curl -k -s https://zdu.binghamton.edu:2345/api/USER_ID | jq .
```
where USER_ID is the portion of your BU email address before the @.
I may reset this server at any time; this means that any changes you have made to your spreadsheet will disappear.
If you access these web services using a browser, you will need to click through the warnings to tell the browser that you are okay using self-signed certificates.

Provided Files

The prj3-sol directory contains a start for your project. It contains the following files:

src/ss-ws.ts

A skeleton file for your project. You should be doing all your development in this file.

config.mjs

A configuration file provided on the command-line when starting the server. It contains information like the port # for the server, as well as specifying the certificates used to run the server.

You should not need to modify this file if you are happy with its contents.

response-envelope.ts

Typescript definitions for the response envelopes.

main.ts

This file provides the complete command-line behavior which is required by your program. It imports the code generated from ss-ws.ts. You should not need to modify this file.

index.mjs

The file invoked on the command-line. It is a trivial wrapper which simply calls main.mjs.

tsconfig.json

A configuration file for typescript. You may modify this file if necessary.

README

A README file which must be submitted along with your project. It contains an initial header which you must complete (replace the dummy entries with your name, B-number and email address at which you would like to receive project-related email). After the header you may include any content which you would like read during the grading of your project.

The test directory contains tests. The web services tests use supertest.

The extras directory contains the following files:

LOG file: A log file which illustrates the operation of the project. Note that this is a fake log file produced non-interactively by running the do-cmd-log.mjs script.
cmds.mjs: The commands used to provide the above LOG.
test1-ss.json: A file used to initialize spreadsheet named test1-ss.

Changes from Project 2

The course lib directory contains a library version of a solution to Project 2. There are some changes:

The spreadsheet DAO is not specific to a single spreadsheet, but can handle multiple spreadsheets; i.e., a DAO is created using simply a database URL and the spreadsheet name ssName is now a parameter to the DAO methods.
All spreadsheet services functions have been changed to methods of a SpreadsheetServices class. The dao parameter to the functions is now an instance variable of the SpreadsheetServices class.

Application Architecture

The overall application architecture is as shown in the following figure:

The lowest level of the software is the DAO implemented in the Project 2 Library. The expressjs app you will be implementing in ss-ws.ts for this project will wrap the ss-services and merely forward HTTP request to the ss-services. Finally, the provided main program in main.js starts a HTTPS server which embeds the app. It is this server which will be accessed by clients on the web.

Hints

The following points are worth noting:

The JSON produced by the web services is not formatted. If the web services are run on the command line using a program like curl, the output can be piped through a program like jq (pre-installed on your VM). If running directly within Chrome (not using some kind of REST client), use a Chrome extension like JSON Formatter.
If you are using curl to test your project and also want to see the response headers on the terminal in additional to jq pretty-printed json, you cannot have curl output the headers to standard output as jq would choke on them. Hence they can be sent to standard error using -D /dev/stderr. Unfortunately, a drawback of this solution is that it becomes impossible to redirect standard error.
To make development easy, it is useful to not have to manually restart the server after each source code change. Automated restarts can be achieved by using a program like nodemon. Once nodemon is installed as a development dependency within your node_modules directory, you can start your server from the prj3-sol directory using something like:
```
    $ npx nodemon --watch ./dist ./index.mjs config.mjs 
```
and the server will be automatically restarted whenever you recompile your TypeScript code to the dist directory.
The above command assumes that spreadsheet data has already been loaded into the database.
As in the previous project, our error-handling convention is that a function indicates an error by returning a Result object having isOk===false with an errors property. We need to convert such errors into an HTTP error envelope as specified in the requirements. That is done by the provided mapResultErrors() utility function.
The code for each handler can be structured as follows:
```
    async function(req: Express.Request, res: Express.Response) {
       try {
         ... usually access a service on app.locals.ssServices
       }
       catch(err) {
        const mapped = mapResultErrors(err);
        res.status(mapped.status).json(mapped);
       }
    }
```
Hence if an error occurs in an intermediate step of processing a request, it is sufficient to merely check for the error and throw:
```
         ...
         const itermediateResult = ...;
         if (!intermediateResult.isOk) {
            throw intermediateResult;
         }
```
with the throw caught by the surrounding try-catch block and converted into an HTTP error.
Almost all validation will be performed by the spreadsheet services available via app.locals.ssServices. If a service returns errors, then those errors can be converted to an HTTP error envelope using code like the above.
If you get a port in use error when attempting to start your server, you probably have an instance of the server already running. Use lsof to discover the pid of that instance and then kill it:
```
    $ lsof -i :2345
    COMMAND     PID ...
    node    1940488 ...
    $ kill 1940488
```

The following steps are not prescriptive in that you may choose to ignore them as long as you meet all project requirements.

Read the project requirements thoroughly. Look at the sample log to make sure you understand the necessary behavior. Review the material covered in class including the express-play example as well as the auth ws slides and auth-ws code.
Look at the expressjs docs. It is probably a good idea to have those docs open in a browser as you work; you will particularly be concerned with the req and res objects.
Your web service implementations will largely consist of accessing the web service parameters and forwarding those parameters onto an appropriate spreadsheet service. Make sure you are familiar with those services from ss-services.ts. A services instance is available within your server as app.locals.ssServices.
Decide on how you will send HTTP requests to your server. You can do so using a command-line HTTP client like curl as in the provided log, an app like Postman, or a browser client like Talend API Tester or yarc.
Set up your prj3-sol branch and submit/prj3-sol as per your previous projects.

Set up your package.json as per your previous project with development dependencies

    @types/chai
    @types/cors
    @types/express
    @types/mocha
    @types/node
    @types/supertest
    chai
    mocha
    mongodb-memory-server
    nodemon
    shx
    supertest
    typescript

and runtime dependencies

    body-parser
    cors
    express
    http-status

as well as the following three cs544 library dependencies as runtime dependencies:

    https://sunybing:sunybing@zdu.binghamton.edu
          /cs544/lib/cs544-js-utils-0.0.1.tgz
    https://sunybing:sunybing@zdu.binghamton.edu
          /cs544/lib/cs544-node-utils-0.0.1.tgz
    https://sunybing:sunybing@zdu.binghamton.edu
          /cs544/lib/cs544-prj2-sol-0.0.1.tgz

Create your own certificate for your VM by running the provided gen-localhost-cert.sh script:
```
    $ ~/cs544/bin/gen-localhost-cert.sh
```
This will generate the certificate files in ~/tmp/localhost-certs. Ignore the characters output on the terminal.

After compiling your project using tsc, you should be able to run the project with a usage message.

    $ ./index.mjs
    usage: index.mjs CONFIG_MJS [SS_JSON_PATH...]
    $ ./index.mjs config.mjs
    listening on port 2345
    ^C       #stop server
    $

You can use the following command to start the server while replacing all data for the spreadsheet test1-ss with the data in test1-ss.json

    $ ./index.mjs config.mjs \
         ~/cs544/projects/prj3/extras/test1-ss.json

Running npm test should result in failing tests.

Quick Sanity Checks: To check whether you have everything installed correctly, try the following:

Access / on the server:

	 # start server in background
	 $ ./index.mjs config.mjs &
	 [1] NNNNNN
	 $ listening on port 2345

	 # GET /
	 $ curl -s -k https://localhost:2345 | jq .
	 {
	   "status": 404,
  	   "errors": [
	     {
	       "options": {
                 "code": "NOT_FOUND"
               },
      	       "message": "GET not supported for /"
             }
           ]
         }

	 # bring server back to foreground
	 $ fg
	 ./index.mjs config.mjs
	 ^C    #stop server
	 $

You should get the 404 error as the provided code does not provide any route for /.

Replace the XXX entries in the README template and commit your project to github.
Open up ss-ws.mjs in an editor. Look at the comments in the file. Note that it contains the top-level exported function, a function for setting up routes, default handlers for 404 and 500 errors and utility functions.
Implement the Get-Cell web service for reading a cell in a spreadsheet. Set up a suitable route in the router. The handler should be a simple wrapper around the query() service provided by app.locals.ssServices. Extract the spreadsheet name and cellId from req.param. Make sure you check for errors and convert them to HTTP errors using the procedure outlined earlier.
Test your implementation by starting your server using the provided test1-ss.json data:
```
    $ ./index.mjs config.mjs ~/cs544/projects/prj3/extras/test1-ss.json
```
and then use curl to send it a suitable request:
```
    curl -s -k -D /dev/stderr \
       'https://localhost:2345/api/test1-ss/a4' \
    | jq .
```
The command pipes the output through jq . to see the JSON pretty-printed. The -D /dev/stderr allows seeing the response headers on standard error.
Implement the Set-Cell web service. The handler should be triggered on a PATCH route and will require accessing the expr query parameter using req.query.expr. It will simply forward the request over to the evaluate() method on the spreadsheet services app.locals.ssServices.
Implement the Copy-Cell service by modifying your handler from the previous step to allow a srcCellId query parameter (instead of the expr query parameter). This kind of request should be forwarded to the copy() method on the spreadsheet services.
Ensure that your handler from the previous step validates its query parameters: i.e. it should check for the presence of exactly one of the query parameters expr or srcCellId.
You should now be able to pass all the tests provided for describe('cells GET and PATCH web services', ...).
Implement the Delete-Cell web service.
You should now be able to pass all the tests provided for describe('cells DELETE web service', ...).
Implement the Clear-Spreadsheet web service.
You should now be able to pass all the tests provided for describe('spreadsheet DELETE web service', ...).
Implement the Load-Spreadsheet web service.
You should now be able to pass all the tests provided for describe('spreadsheet PUT web service', ...).
Implement the Get-Spreadsheet web service.
You should now be able to pass all the tests provided for describe('spreadsheet GET web service', ...).
Test using the command-line as in the provided LOG.
You can produce the log using the do-cmd-log.mjs script. Make sure you have a clean database by restarting your server with the test1-ss.json file and then use the do-cmd-log.mjs script:
```
    $ ~/cs544/bin/do-cmd-log.mjs ~/cs544/projects/prj3/extras/cmds.mjs
```
This output will differ from the provided LOG in the Date headers and possibly in the ETag headers. You can filter those out using grep -v:
```
    $ cat ~/cs544/projects/prj3/extras/LOG \
        | grep -v '^Date:' \
        | grep -v '^ETag:' \
	> ~/tmp/t1
    $ ~/cs544/bin/do-cmd-log.mjs ~/cs544/projects/prj3/extras/cmds.mjs \	
        | grep -v '^Date:' \
        | grep -v '^ETag:' \
	> ~/tmp/t2
    $ diff ~/tmp/t1 ~/tmp/t2
```
If the last command returns silently, then your output matches the provided LOG.
Iterate until you meet all requirements.
Clean up:
1. Remove any .only or .skip you may have added to the tests.
2. Comment out or remove any added debugger lines.
3. Remove any print statements so that the tests run without producing extraneous output.

It is a good idea to commit and push your project periodically whenever you have made significant changes.

Submit as per your previous project. Before submitting, please update your README to document the status of your project:

Document known problems. If there are no known problems, explicitly state so.
Anything else which you feel is noteworthy about your submission.

If you want to make sure that your github submission is complete, clone your github repo into a new directory, say ~/tmp. You should then be able to do a npm ci to build and run your project.