Core 1830 Docs Migration: Custom Functions 1 of 2

docs/custom-functions/create-project.md

@@ -0,0 +1,41 @@
+# Create a Project
+One easy way to manage Custom Functions is through [HarperDB Studio](https://harperdb.io/docs/harperdb-studio/manage-functions/). You can read more about [managing Custom Functions through the HarperDB Studio here](https://harperdb.io/docs/harperdb-studio/manage-functions/).
+
+To manually create a project, you have three options:
+
+
+
+1. **use the add_custom_function_project operation**
+
+   This operation creates a new project folder, and populates it with templates for the routes, helpers, and static subfolders.
+
+```json
+{
+   "operation": "add_custom_function_project",
+   "project": "dogs"
+}
+```
+
+
+2. **clone our public gitHub project template**
+
+   _This requires a local installation. Remove the .git directory for a clean slate of git history._
+
+
+```bash
+> git clone https://github.com/HarperDB/harperdb-custom-functions-template.git ~/hdb/custom_functions/dogs
+```
+
+
+3. **create a project folder in your Custom Functions root directory** and **initialize**
+
+   _This requires a local installation._
+
+
+```bash
+> mkdir ~/hdb/custom_functions/dogs
+```
+
+```bash
+> npm init
+```

docs/custom-functions/define-helpers.md

@@ -0,0 +1,32 @@
+# Define Helpers
+
+Helpers are functions for use within your routes. You may want to use the same helper in multiple route files, so this allows you to write it once, and include it wherever you need it.
+
+
+
+* To use your helpers, they must be exported from your helper file. Please use any standard export mechanisms available for your module system. We like ESM, ECMAScript Modules. Our example below exports using `module.exports`.
+
+* You must import the helper module into the file that needs access to the exported functions. With ESM, you'd use a `require` statement. See [this example](define-routes.md#custom-preValidation-hooks) in Define Routes.
+
+
+Below is code from the customValidation helper that is referenced in [Define Routes](define-routes.md). It takes the request and the logger method from the route declaration, and makes a call to an external API to validate the headers using fetch. The API in this example is just returning a list of ToDos, but it could easily be replaced with a call to a real authentication service.
+
+
+```javascript
+const customValidation = async (request,logger) => {
+    let response = await fetch('https://jsonplaceholder.typicode.com/todos/1', { headers: { authorization: request.headers.authorization } });
+    let result = await response.json();
+
+   /*
+   *  throw an authentication error based on the response body or statusCode
+   */
+   if (result.error) {
+      const errorString = result.error || 'Sorry, there was an error authenticating your request';
+      logger.error(errorString);
+      throw new Error(errorString);
+   }
+   return request;
+};
+
+module.exports = customValidation;
+```

docs/custom-functions/define-routes.md

@@ -0,0 +1,127 @@
+# Define Routes
+
+HarperDB’s Custom Functions is built on top of [Fastify](https://www.fastify.io/), so our route definitions follow their specifications. Below is a very simple example of a route declaration.
+
+
+
+Route URLs are resolved in the following manner:
+
+* [**Instance URL**]:[**Custom Functions Port**]/[**Project Name**]/[**Route URL**]
+
+* The route below, within the **dogs** project, with a route of **breeds** would be available at **http://localhost:9926/dogs/breeds**.
+
+
+In effect, this route is just a pass-through to HarperDB. The same result could have been achieved by hitting the core HarperDB API, since it uses **hdbCore.preValidation** and **hdbCore.request**, which are defined in the “helper methods” section, below.
+
+
+
+```javascript
+module.exports = async (server, { hdbCore, logger }) => {
+    server.route({
+        url: '/',
+        method: 'POST',
+        preValidation: hdbCore.preValidation,
+        handler: hdbCore.request,
+    })
+}
+```
+
+
+## Custom Handlers
+
+For endpoints where you want to execute multiple operations against HarperDB, or perform additional processing (like an ML classification, or an aggregation, or a call to a 3rd party API), you can define your own logic in the handler. The function below will execute a query against the dogs table, and filter the results to only return those dogs over 4 years in age.
+
+
+
+**IMPORTANT: This route has NO preValidation and uses hdbCore.requestWithoutAuthentication, which- as the name implies- bypasses all user authentication. See the security concerns and mitigations in the “helper methods” section, below.**
+
+
+
+```javascript
+module.exports = async (server, { hdbCore, logger }) => {
+    server.route({
+        url: '/:id',
+        method: 'GET',
+        handler: (request) => {
+            request.body= {
+                operation: 'sql',
+                sql: `SELECT * FROM dev.dog WHERE id = ${request.params.id}`
+            };
+
+          const result = await hdbCore.requestWithoutAuthentication(request);
+          return result.filter((dog) => dog.age > 4);
+        }
+    });
+}
+```
+
+## Custom preValidation Hooks
+The simple example above was just a pass-through to HarperDB- the exact same result could have been achieved by hitting the core HarperDB API. But for many applications, you may want to authenticate the user using custom logic you write, or by conferring with a 3rd party service. Custom preValidation hooks let you do just that.
+
+
+
+Below is an example of a route that uses a custom validation hook:
+
+```javascript
+const customValidation = require('../helpers/customValidation');
+
+module.exports = async (server, { hdbCore, logger }) => {
+    server.route({
+        url: '/:id',
+        method: 'GET',
+        preValidation: (request) => customValidation(request, logger),
+        handler: (request) => {
+            request.body= {
+                operation: 'sql',
+                sql: `SELECT * FROM dev.dog WHERE id = ${request.params.id}`
+            };
+
+         return hdbCore.requestWithoutAuthentication(request);
+        }
+    });
+}
+```
+
+
+Notice we imported customValidation from the **helpers** directory. To include a helper, and to see the actual code within customValidation, see [Define Helpers](define-helpers.md).
+
+## Helper Methods
+When declaring routes, you are given access to 2 helper methods: hdbCore and logger.
+
+
+
+**hdbCore**
+
+hdbCore contains three functions that allow you to authenticate an inbound request, and execute operations against HarperDB directly, by passing the standard Operations API.
+
+
+
+* **preValidation**
+
+   This takes the authorization header from the inbound request and executes the same authentication as the standard HarperDB Operations API. It will determine if the user exists, and if they are allowed to perform this operation. **If you use the request method, you have to use preValidation to get the authenticated user**.
+
+* **request**
+
+  This will execute a request with HarperDB using the operations API. The `request.body` should contain a standard HarperDB operation and must also include the `hdb_user` property that was in `request.body` provided in the callback.
+
+* **requestWithoutAuthentication**
+
+   Executes a request against HarperDB without any security checks around whether the inbound user is allowed to make this request. For security purposes, you should always take the following precautions when using this method:
+
+  * Properly handle user-submitted values, including url params. User-submitted values should only be used for `search_value` and for defining values in records. Special care should be taken to properly escape any values if user-submitted values are used for SQL.
+
+
+**logger**
+
+This helper allows you to write directly to the Custom Functions log file, custom_functions.log. It’s useful for debugging during development, although you may also use the console logger. There are 5 functions contained within logger, each of which pertains to a different **logging.level** configuration in your harperdb-config.yaml file.
+
+
+* logger.trace(‘Starting the handler for /dogs’)
+
+* logger.debug(‘This should only fire once’)
+
+* logger.warn(‘This should never ever fire’)
+
+* logger.error(‘This did not go well’)
+
+* logger.fatal(‘This did not go very well at all’)

docs/custom-functions/host-static.md

@@ -0,0 +1,27 @@
+# Host A Static Web UI
+We’ve configured the Custom Functions server to automatically serve any static UI files it finds in a project’s **/static** folder.
+
+
+In order to serve your static UI correctly, your UI project must meet the following requirements:
+
+* Have an index file located at **/static/index.html**
+
+* Correctly path any other files relative to index.html
+
+* If your app makes use of client-side routing, it must have **[project_name]** as its base (`basename` for react-router, `base` for vue-router, etc.):
+
+```javascript
+<Router basename="/dogs">
+    <Switch>
+        <Route path="/care" component={CarePage} />
+        <Route path="/feeding" component={FeedingPage} />
+    </Switch>
+</Router>
+```
+
+
+Supporting files, like css, js, and images may be located within subfolders or at the root of the **/static** folder, whichever you prefer.
+
+
+
+If you’re using a framework (React, Vue, etc.), we recommend dropping the output of your build process (your `dist` or `build` folder) directly into your project’s **/static** folder. The output of the build process is usually optimized and compressed, and will be more performant than raw source code.

docs/custom-functions/index.md

@@ -1 +1,24 @@
 # Custom Functions
+
+Custom functions are a key part of building a complete HarperDB application. It is highly recommended that you use custom functions as the primary mechanism for your application to access your HarperDB database. Using custom functions gives you complete control over the accessible endpoints, how users are authenticated and authorized, what data is accessed from the database, and how it is aggregated and returned to users.
+
+* Add your own API endpoints to a standalone API server inside HarperDB
+
+* Use HarperDB Core methods to interact with your data at lightning speed
+
+* Custom Functions are powered by Fastify, so they’re extremely flexible
+
+* Manage in HarperDB Studio, or use your own IDE and Version Management System
+
+* Distribute your Custom Functions to all your HarperDB instances with a single click
+
+---
+* [Requirements and Definitions](requirements-definitions.md)
+
+* [Create A Project](create-project.md)
+
+* [Define Routes](define-routes.md)
+
+* [Define Helpers](define-helpers.md)
+
+* [Host a Static UI](host-static.md)

docs/custom-functions/requirements-definitions.md

@@ -0,0 +1,73 @@
+# Requirements And Definitions
+Before you get started with Custom Functions, here’s a primer on the basic configuration and the structure of a Custom Functions Project.
+
+## Configuration
+Custom Functions are configured in the harperdb-config.yaml file located in the operations API root directory (by default this is a directory named `hdb` located in the home directory of the current user). Below is a view of the Custom Functions' section of the config YAML file, plus descriptions of important Custom Functions settings. 
+
+```yaml
+customFunctions:
+  enabled: true
+  network:
+    cors: true
+    corsAccessList:
+      - null
+    headersTimeout: 60000
+    https: false
+    keepAliveTimeout: 5000
+    port: 9926
+    timeout: 120000
+  nodeEnv: production
+  root: ~/hdb/custom_functions
+  tls:
+    certificate: ~/hdb/keys/certificate.pem
+    certificateAuthority: ~/hdb/keys/ca.pem
+    privateKey: ~/hdb/keys/privateKey.pem
+```
+
+* **`enabled`**
+  A boolean value that tells HarperDB to start the Custom Functions server. Set it to **true** to enable custom functions and **false** to disable. `enabled` is `true` by default.
+
+* **`network.port`**
+  This is the port HarperDB will use to start a standalone Fastify Server dedicated to serving your Custom Functions’ routes.
+
+* **`root`**
+  This is the root directory where your Custom Functions projects and their files will live. By default, it’s in your \<ROOTPATH>, but you can locate it anywhere--in a developer folder next to your other development projects, for example.
+
+_Please visit our [configuration docs]() for a more comprehensive look at these settings._
+
+## Project Structure
+**project folder**
+
+The name of the folder that holds your project files serves as the root prefix for all the routes you create.  All routes created in the **dogs** project folder will have a URL like this: **https://my-server-url.com:9926/dogs/my/route**. As such, it’s important that any project folders you create avoid any characters that aren’t URL-friendly. You should avoid URL delimiters in your folder names.
+
+
+**/routes folder**
+
+Files in the **routes** folder define the requests that your Custom Functions server will handle. They are [standard Fastify route declarations](https://www.fastify.io/docs/latest/Routes/), so if you’re familiar with them, you should be up and running in no time. The default components for a route are the url, method, preValidation, and handler.
+
+```javascript
+module.exports = async (server, { hdbCore, logger }) => {
+    server.route({
+        url: '/',
+        method: 'POST',
+        preValidation: hdbCore.preValidation,
+        handler: hdbCore.request,
+    });
+}
+```
+
+**/helpers folder**
+
+These files are JavaScript modules that you can use in your handlers, or for custom `preValidation` hooks. Examples include calls to third party Authentication services, filters for results of calls to HarperDB, and custom error responses. As modules, you can use standard import and export functionality.
+
+```javascript
+"use strict";
+
+const dbFilter = (databaseResultsArray) => databaseResultsArray.filter((result) => result.showToApi === true);
+
+module.exports = dbFilter;
+```
+
+**/static folder**
+
+If you’d like to serve your visitors a static website, you can place the html and supporting files into a directory called **static**. The directory must have an **index.html** file, and can have as many supporting resources as are necessary in whatever subfolder structure you prefer within that **static** directory.

docs/harperdb-studio/index.md

@@ -5,7 +5,7 @@ HarperDB Studio is the web-based GUI for HarperDB. Studio enables you to adminis
 
 ---
 ## How does Studio Work?
-While HarperDB Studio is web based and hosted by us, all database interactions are performed locally. The HarperDB Studio loads in your browser, at which point you login to your HarperDB instances. Credentials are stored in your browser cache and are not transmitted back to HarperDB. All database interactions are made via the HarperDB Operations API directly from your browser to your instance.
+While HarperDB Studio is web based and hosted by us, all database interactions are performed on the HarperDB instance the studio is connected to. The HarperDB Studio loads in your browser, at which point you login to your HarperDB instances. Credentials are stored in your browser cache and are not transmitted back to HarperDB. All database interactions are made via the HarperDB Operations API directly from your browser to your instance.
 
 ## What type of instances can I manage?
 HarperDB Studio enables users to manage both HarperDB Cloud instances and privately hosted instances all from a single UI. All HarperDB instances feature identical behavior whether they are hosted by us or by you.

docs/security/basic-auth.md

@@ -18,6 +18,8 @@ A header is added to each HTTP request.  The header key is **“Authorization”
 
 Below is a code sample from HarperDB Studio.  On Line 14 you will see where we are adding the authorization header to each request. This needs to be added for each and every HTTP request for HarperDB.
 
+_Note: This function uses btoa. Learn about [btoa here](https://developer.mozilla.org/en-US/docs/Web/API/btoa)._
+
 ```javascript
 function callHarperDB(call_object, operation, callback){
 

docs/security/users-and-roles.md

@@ -20,17 +20,23 @@ Role permissions in HarperDB are broken into two categories – permissions arou
 
 
 **Built-In Roles**
-There are two built-in roles within HarperDB. See full breakdown of operations restricted to only super_user roles [here](https://harperdb.io/docs/security/users-roles/).
 
-* `super_user` – this role provides full access to all operations and methods within a HarperDB instance, this can be considered the admin role.
+There are three built-in roles within HarperDB. See full breakdown of operations restricted to only super_user roles [here](https://harperdb.io/docs/security/users-roles/).
+
+* `super_user` - This role provides full access to all operations and methods within a HarperDB instance, this can be considered the admin role.
 
   * This role provides full access to all Database Definition operations and the ability to run Database Manipulation operations across the entire database schema with no restrictions. 
-* `cluster_user` – this role is an internal system role type that is managed internally to allow clustered instances to communicate with one another.
+
+* `cluster_user` - This role is an internal system role type that is managed internally to allow clustered instances to communicate with one another.
 
   * This role is an internally managed role to facilitate communication between clustered instances.
 
+* `structure_user` - This role provides specific access for creation and deletion of data.
+
+  * When defining this role type you can either assign a value of true which will allow the role to create and drop schemas & tables. Alternatively the role type can be assigned a string array. The values in this array are schemas and allows the role to only create and drop tables in the designated schemas.
 
 **User-Defined Roles**
+
 In addition to built-in roles, admins (i.e. users assigned to the super_user role) can create customized roles for other users to interact with and manipulate the data within explicitly defined tables and attributes.
 
 * Unless the user-defined role is given `super_user` permissions, permissions must be defined explicitly within the request body JSON.