Initial Commit

Author: Matthew Forrester

.gitignore

@@ -0,0 +1,2 @@
+/target
+**/*.rs.bk

Cargo.lock

@@ -0,0 +1,12 @@
+[package]
+name = "wv_linewise"
+version = "0.1.0"
+authors = ["Matthew Forrester <matthew.forrester@speechmarks.com>"]
+edition = "2018"
+
+[dependencies]
+web-view = { version = "0.6.2", features = ["edge"] }
+serde_json = { version = "1.0", default-features = false, features = ["alloc"] }
+serde_derive = "1.0.105"
+serde = { version = "1.0", features = ["derive"] }
+clap = "2.33.0"

Cargo.toml

@@ -0,0 +1,198 @@
+# WV Linewise
+
+## The Potential User
+
+You're a software developer and UNIX system administrator.
+
+You loooove the command line.
+
+You have probably looked at GNU Plot because it could draw graphs on the terminal but feel it's output isn't suitable for you.
+
+Also you write your SQL in VIM and have it integrated with TMUX to send your query to `psql` you get mildly annoyed because of the lack of horizontal scrolling.
+
+Maybe you use i3...
+
+You write software for a living, sometimes you write great code. Your code is sometimes deployed into big integrated projects but sometimes it's only about getting data from A to B, a one off and while it works, you wonder if `BLAH BLAH | SOMETHING | X | jq BLAH | Y | Z` might have worked, and may have been faster if only you could add some interactivity to X.
+
+You need one foot in both worlds... how...
+
+## What it does
+
+[Boscop's Web-View](https://github.com/Boscop/web-view) has provided us with a lightweight library for showing HTML/CSS/JS within a window, using the OS's standard browser. If you know HTML/CSS/JS but shudder at the weight of Electron, this may be your thing. As a bonus, it's written in Rust too!
+
+This project provides some Rust functions for streaming STDIN or files into Boscop's Web-View and a higher level TypeScript / JavaScript API to pull them in as a stream of lines. You can also write to STDOUT using this API enabling you to put something like a web page, right in the middle of a UNIX pipeline. I find this quite exciting.
+
+Can you think of any things which you would like to do using UNIX pipes which you think you may want to add some interactivity or graphics to? I can think of many...
+
+## Example
+
+This program could be invoked like the following:
+
+
+    # Prepare lookups.csv
+    echo '"First Name","Telephone Number"' > lookups.csv
+    echo 'Alice,"01922 123 456"' >> lookups.csv
+    echo 'Ben,"0800 100 232"' >> lookups.csv
+    echo 'Jack,"01882 556216"' >> lookups.csv
+
+    # write a file that we will pipe to stdin
+    echo 'Name,Age' > stdin.csv
+    echo 'Ben,21' >> stdin.csv
+    echo 'Alice,31' >> stdin.csv
+    echo 'Jane,27' >> stdin.csv
+
+    cat stdin.csv | wv-linewise \
+        --code tables \
+        --stream this_is_stdin=- \
+        --stream lookups=lookups.tsv
+        --param this_is_stdin='["Name"]' \
+        --param lookups='["First Name"]' \
+        --param request_count=2
+
+
+The data input above would cause the following messages to be sent between WV Linewise and the embedded web page.
+
+    < { "msg": "params" }
+    > { "type": "params", "params": [{ "name": "this_is_stdin", value: "[\"Name\"]"}, {"name": "lookups": "value": "[\"First Name\"]"}] }
+    < {"msg":"streamList"}
+    > {"streams":["this_is_stdin","lookups"],"type":"streamList"}
+    < { "msg": "out", "descriptor": 1, "data": "Line Number,Name,Age,Telephone Number" }
+    < { "msg": "streamStart", "name": "this_is_stdin", "count": 2 }
+    < { "msg": "streamStart", "name": "lookups", "count": 2 }
+    > { "type": "details", "name": "this_is_stdin", "details": { "rewindable": false }
+    > { "type": "details", "name": "lookups", "details": { "rewindable": false }
+    > { "type": "line", "name": "this_is_stdin", "data": "Name,Age" }
+    > { "type": "line", "name": "this_is_stdin", "data": "Ben,21" }
+    > { "type": "paused", "name": "this_is_stdin" }
+    > { "type": "line", "name": "lookups", "data": "\"First Name\",\"Telephone Number\"" }
+    < { "msg": "streamContinue", "name": "this_is_stdin" }
+    > { "type": "line", "name": "lookups", "data": "Alice,\"01922 123 456\"" }
+    > { "type": "paused", "name": "lookups" }
+    > { "type": "line", "name": "this_is_stdin", "data": "Alice,31" }
+    < { "msg": "out", "descriptor": 1, "data": "2,Alice,31,\"01922 123 456\"" }
+    > { "type": "line", "name": "this_is_stdin", "data": "Jane,27" }
+    > { "type": "finished", "name": "this_is_stdin" }
+    < { "msg": "streamContinue", "name": "lookups" }
+    > { "type": "line", "name": "lookups", "data": "Ben,\"0800 100 232\"" }
+    < { "msg": "out", "descriptor": 1, "data": "1,Ben,21,\"0800 100 232"" }
+    > { "type": "line", "name": "lookups", "data": "Jack,\"01882 556216\"" }
+    > { "type": "finished", "name": "lookups" }
+    < { "msg": "exit", "status": 0 }
+
+NOTE: `<` are messages from TypeScript / JavaScript to WV Linewise, `>` are the responses.
+
+WV Linewise will then exit with a status code of 0 and the following data will have already been written to STDOUT:
+
+    Line Number,Name,Age,Telephone Number
+    2,Alice,31,"01922 123 456"
+    1,Ben,21,"0800 100 232"
+
+
+## APIs
+
+There are two TypeScript / JavaScript APIs I created to control the sending / receiving of messages. These are listed below:
+
+### The original Light Wrapper API
+
+If for whatever reason you don't want to use the Buffer API, the original API is a light weight message / event based API. In writing it I was merely trying to add some type safety over the raw messages. See the example below
+
+```typescript
+import { RawWvLinewise, WvLinewise, RESPONSE_TYPE, MessageErrorResponse, ErrorResponse, ParamsResponse, LineResponse, PausedResponse } from "wv-linewise-js-lib";
+
+async function processLightWeight() {
+
+let lineCount = 0;
+const wvl: WvLinewise = new WvLinewise(new RawWvLinewise(external as any));
+
+// Upon error, just raise it so it's caught by the global error handler.
+wvl.on(RESPONSE_TYPE.MESSAGE_ERROR, (msg: MessageErrorResponse) => {
+    throw new Error(`MSG ERROR: ${JSON.stringify(msg)}`)
+});
+
+// Upon error, just raise it so it's caught by the global error handler.
+wvl.on(RESPONSE_TYPE.ERROR, (msg: ErrorResponse) => {
+    throw new Error(`MSG ERROR: ${JSON.stringify(msg)}`)
+});
+
+// Request the parameters the user passed in on the command line
+function getParams(wvl: WvLinewise): Promise<ParamsResponse> {
+    return new Promise((resolve) => {
+        let f = (resp: ParamsResponse) => {
+            resolve(resp);
+        };
+        wvl.once(RESPONSE_TYPE.PARAMS, f);
+        wvl.requestParams();
+    });
+}
+
+function getRequestQuantity(paramsResponse: ParamsResponse): number {
+    for (let p of paramsResponse.params) {
+        if (p.name == "quantity") {
+            return parseInt(p.value, 10);
+        }
+    }
+    return 1000;
+}
+
+// Because all of our code is blocking (we're not waiting for animations etc)
+// we're going to have processed the data immediately, so when WV Linewise
+// pauses we can just start it right up again.
+wvl.on(RESPONSE_TYPE.PAUSED, (resp: PausedResponse) => {
+    if (resp.name == "in") {
+        wvl.streamContinue("in");
+    }
+});
+
+// This function will get fired on every line, with the line that came from
+// the "in" stream, which could be STDIN or a file.
+wvl.on(RESPONSE_TYPE.LINE, (resp: LineResponse) => {
+    if (resp.name == "in") {
+        lineCount = lineCount + 1;
+    }
+    document.body.innerText = `The file has ${lineCount} lines`
+});
+
+
+// Start WV Linewise processing lines
+wvl.streamStart("in", getRequestQuantity(await getParams(wvl)));
+
+}
+
+processLightWeight();
+```
+
+### The buffer API
+
+The WvLinewiseBuffer will allow you to disregard the messages for the purposes of reading the streams. It uses a low watermark and a quantity to request (3rd and 4th parameters) to try and make sure there's always lines available. Because it's built upon Promises it can handle situations where the buffer is empty and not fail.
+
+```typescript
+async function processBuffer(wvl: WvLinewise) {
+
+    let buffer = new WvLinewiseBuffer(wvl, "in", 100, 200);
+    let line: string|null = "";
+    let lineCount = 0;
+
+    while (line !== null) {
+        line = await buffer.shift();
+        if (line === null) {
+            continue;
+        }
+        document.body.innerText = `The file has ${lineCount} lines and the last line was ${line}`;
+    }
+}
+
+const wvl: WvLinewise = new WvLinewise(new RawWvLinewise(external as any));
+processBuffer(wvl);
+```
+
+## Example Applications
+
+### [Discover Types](./examples/discover-types)
+
+#### About
+DiscoverTypes is an application for identifying the types of fields within a CSV file. It does this by comparing every cell against every regular expression the user has supplied. Because the CSV file could be HUGE it does not load the whole thing into memory but inspects it line by line as it passes through.
+
+#### Screenshot
+
+![What it looks like](./img/screenshot.png)
+

README.md

@@ -0,0 +1,23 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# production
+/build
+
+# misc
+.DS_Store
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*

examples/discover-types/.gitignore

@@ -0,0 +1,94 @@
+# Discover Types
+
+## About
+
+DiscoverTypes is an application for identifying the types of fields within a CSV file. It does this by comparing every cell against every regular expression the user has supplied. Because the CSV file could be HUGE it does not load the whole thing into memory but inspects it line by line as it passes through.
+
+## Screenshot
+
+![What it looks like](./img/screenshot.png)
+
+## The Output
+
+The following is an example of the CSV output:
+
+| Column      | Types                                     | Count   | Total Record Count    |
+| ------------|-------------------------------------------|---------|-----------------------|
+| salesPerson | ["String"]"                               | 13      | 13                    |
+| date        | ["String","Full US Date","Full UK Date"]" | 8       | 13                    |
+| date        | ["NULL"]"                                 | 1       | 13                    |
+| date        | ["String","Full UK Date"]"                | 4       | 13                    |
+| orderId     | ["Integer","String"]"                     | 13      | 13                    |
+| item        | ["String"]"                               | 13      | 13                    |
+| cost        | ["String"]"                               | 13      | 13                    |
+| price       | ["Integer","String"]"                     | 10      | 13                    |
+| price       | ["String"]"                               | 3       | 13                    |
+| profit      | ["String"]"                               | 13      | 13                    |
+
+Every cell within a CSV could be interpreted as 0 or more different types. Generally speaking I would use the regular expression `^$` (meaning a zero length string) to be `NULL` and `.` (at least one character in length) to be a String. This means that the integer `4` might be an integer (`^\-[0-9]+`) but could also a string. This is completely correct as you might have user input of a string type, which users tend to write integers in, an example of this may be local phone numbers, which are often just numbers, but sometimes include other characters, so are forced to be strings.
+
+### Why we don't prefer one type over another
+
+Imagine you had 14 million lines and one column was full of dates and we preferred US dates format over the UK dates. The software would tell us we had 14 million (minus one) US dates and 1 UK date and you may conclude that the column is US dates with one error... but if all the dates were before the 12th of the month except __that__ one, you'd probably be wrong.
+
+This situation is actually shown in a much more simple form above. You can see that the date column must be one of the following:
+
+ * A "Full US Date" and therefore has 5 records in error.
+ * A "String" and therefore has 1 record in error.
+ * A "Full UK Date" and therefore has 1 record in error.
+ * **A "String" with null allowed and therefore has 0 record in error.**
+ * **A "Full UK Date" with null allowed and therefore has 0 record in error.**
+
+Given the output above it should be relatively trivial to write software, to figure out that the date column is actually a UK Date which allows nulls. But the caveat is that software needs to know to prefer "Full UK Date" over "String" which this software does not.
+
+I may well add code and columns to:
+
+ * List out what the zero error candidates are. This would get you down to just "String" and "Full UK Date" in the above example.
+ * Add at least one example for each row in the table above.
+
+## Usage
+
+```shell
+cat test-data/burger-shop.csv | wv-linewise --code index.html \
+  --stream in=- \
+  --stream types=types.csv \
+  --param 'Full US Date'='^0*[0-9][0-2]?/[0-9]+/[0-9]{4}$' \
+  --param 'Full UK Date'='^[0-9]+/0*[0-9][0-2]?/[0-9]{4}$' \
+  --param mode='continuous'
+```
+
+Because DiscoverTypes is built upon WV Linewise it's command line interface is the interface from WV Linewise... well I could wrap the program in a BASH/BAT file to make it more slick, but DiscoverTypes is actually only an example application for WV Linewise so I will not.
+
+### The "in" stream ( required )
+
+The "in" stream is where the actual CSV you want to inspect comes from. It could be huge...
+
+If you use the special value `-` WV Linewise will read STDIN, otherwise this will be taken to be a filename and that file will be read.
+
+### The "types" stream (optional)
+
+The "type" stream is a CSV with the following structure.
+
+| name    | regexp    |
+|---------|-----------|
+| Integer | ^-?[0-9]+ |
+| String  | .         |
+| NULL    | ^$        |
+
+The `name` is a types name and `regexp` is a JavaScript regular expression without the enclosing forward slashes.
+
+### The "type" parameters (optional)
+
+The type parameters are added to the "types" stream above with the `name` and the `regexp` taking the same form. Because WV Linewise has a singular `--param` command line argument written in the form `--param 'name=value`, which is moderately too verbose but at least quite specific.
+
+The name must not conflict with any other parameter names otherwise it will need to can be specified in full, for example such as `--param 'type=Full UK Date=^[0-9]+/0*[0-9][0-2]?/[0-9]{4}$'`. Type parameters names also cannot include the `=` symbol.
+
+### The "mode" parameter (optional)
+
+I think the ease of adding interactive elements within UNIX pipelines is one of the nicest parts of WV Linewise, but equally I would find it mildly annoying to page through over a million rows.
+
+To deal with this I have introduced a "mode" to Discover Types which can take one of three values:
+
+ * `manual` The default mode is "manual" and in this mode you will have to press "More" on the user interface to page through the data.
+ * `continuous` This mode will page automatically and will stop at the end, waiting for you to press the "Exit" button,
+ * `continuous-and-exit` This is same as `continuous` except that it exits automatically at the end.

examples/discover-types/README.md

@@ -0,0 +1,12 @@
+var fs = require("fs")
+var inlineAssets = require("inline-assets")
+var content = fs.readFileSync("build/index.html", "utf8")
+content = inlineAssets("index.html", "build/index.html", content, {
+    verbose: false,
+    htmlmin: false,
+    cssmin:  false,
+    jsmin:   false,
+    pattern: [ ".+" ],
+    purge:   false
+})
+fs.writeFileSync("index.html", content, "utf8")

examples/discover-types/build.js

@@ -0,0 +1,7 @@
+#!/bin/bash
+set -euo pipefail
+IFS=$'\n\t'
+
+rm -rf build
+npm run-script build && sed -i 'sJ="/J="Jg' build/index.html && node build.js
+