From f4e21c168aa1bfd69afcb76543d9a5e79c0113bc Mon Sep 17 00:00:00 2001 From: Damon Sicore Date: Fri, 14 Feb 2020 11:12:31 -0600 Subject: [PATCH 01/13] Adding an NEP for NEAR Shell enhancements (#30) --- text/0000-near-shell.md | 191 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 text/0000-near-shell.md diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md new file mode 100644 index 000000000..fec87e138 --- /dev/null +++ b/text/0000-near-shell.md @@ -0,0 +1,191 @@ +- Proposal Name: near-shell +- Start Date: 2020-02-03) +- NEP PR: [nearprotocol/neps#0000](https://github.com/nearprotocol/neps/pull/0000) +- Issue(s): link to relevant issues in relevant repos (not required). + +# Summary +[summary]: #summary + +NEAR Shell is one of the primary tools NEAR developers must use to develop and interact with NEAR networks and smart contracts. Today, NEAR Shell contains the base commands for account creation and deletion, login, viewing account state and keys, sending tokens, creating staking transactions, and building, deploying, and calling smart contracts. There are more opportunities to integrate core NEAR features to expose them directly to users. This proposal is to enhance NEAR Shell to represent all major functionality of NEAR. For example, validator node management, network selection, native OS key management and integration, smart contract development (including testing, and interaction) could be added. + +NEAR Shell is an excellent tool to attract and engage new NEAR developers, both core and contract level, and if it contains all the required features to select and configure and deploy NEAR networks, nodes, crypto assets (i.e., keys), accounts, and wallets, it can be the ideal single-entry point for those potential new hackers. + +# Motivation +[motivation]: #motivation + +These enhancements are intended to simplify and enhance developer and user productivity by providing a single command-line tool which exercises all major features of NEAR. As each major piece of NEAR functionality is added to Near Shell, we should see utilization of each of those features increase. Additionally, community mindshare and understanding of NEAR should increase as NEAR Shell will contain detailed help documentation for each major feature with clear examples of use, just like Linux man pages and/or other major CLI tools which have integrated help for each command. + + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +Using NEAR Shell, users can configure accounts, networks, and an instance of a server validator node on the host machine the shell is running. An instance of NEAR Shell loads its state from the configuration directory and files stored in the default location: `~/.near`. Once the configuration is loaded, the active account and network are selected automatically either from the previously stored configuration or the defaults as shipped with NEAR Shell. + +## NEAR Shell Top-Level Commands + +### `near network ` + +This command category is used to select and configure NEAR networks for a NEAR Shell user. This category manipulates the user's `~/.near` configuration to allow a user to specify NEAR network details via the CLI instead of manually editing config files. + +Sub-commands for `network` might include: + +* `near network list` : Display the current list of networks for the current instance of NEAR Shell. +* `near network status` : Show the user's current network configuration. +* `near network select` : Select a NEAR network as the default network for subsequent NEAR Shell commands as well as the network configuration for a local server validator node managed by this instance of NEAR Shell. +* `near network add` : Add a network to the users's `~/.near` config files. +* `near network remove` : Remove a network from tthe `~/.near` config files. +* `near network monitor` : Interactively monitor one or more NEAR networks in a curses-like interface updated in a specified interval, default of 1s. +* `near network help` : Display help on network subcommands. + +### `near account ` + +This category is used to create, select, and configure accounts on NEAR networks for a given NEAR Shell user. + +* `near account list` : Display list of accounts configured on local host. +* `near account status` : Display status of active or specified account. +* `near account create` : Create a new account or sub-account. +* `near account delete` : Delete an account or sub-account. +* `near account select` : Select an account from the list of locally configured accounts as the active account. +* `near account login` : Log in the current active or specified account. +* `near account logout` : Log out the current active or specified account. +* `near account help` : Display help on account subcommands. + +### `server` + +These commands are used to administer a NEAR server validator node. Currently, only one server per NEAR Shell is proposed; however, multiple servers per physical host should be considered. + +* `near server status` : Display status of NEAR validator server on current host. +* `near server start` : Start NEAR validator server on current host. +* `near server stop` : Stop NEAR validator server on current host. +* `near server monitor` : Interactively display NEAR validator server status on current host. +* `near server tail` : Tail the log of the NEAR validator server on current host. +* `near server help` : Display help on server subcommands. + +### `near tokens ` + +* `near tokens send` : Send tokens to another NEAR account. +* `near tokens status` : Display tokens in wallet. +* `near tokens stake` +* `near tokens help` + +### `near key ` + +### `near contract ` +* `near contract list` +* `near contract status` +* `near contract add` +* `near contract remove` +* `near contract build` +* `near contract deploy` +* `near contract call` + +### `near config ` + + +### User Stories +For user-facing NEPs this section should focus on user stories. + +These enhancements to near-shell aim to increase user and developer adoption by centralizing and optimizing all interactions with NEAR functionality. Near Shell today is a thin and lightweight set of features which enable the use of each NEAR product offering. Specifically, Near Shell is required to use NEAR products today. For example, to stake, send tokens, create and delete accounts, view public keys, build and deploy smart contracts, call smart contract methods, and log in through NEAR Protocol's wallet. Each of these features are critical to all NEAR products. + +If NEAR Shell is required, it makes sense to enhance NEAR Shell to include other features critical to user and developer interactions. The commands proposed above enhance NEAR Shell to include validator node deployment and control for both local and distributed development, i.e., using a local network for offline development vs. testnet as a live network used by others, and to be able to select configure and select networks. + +Here are a few user stories to demonstrate the utility of these enhancements. + +#### Creating and Configuring Networks + +When a user or developer wants to try out a simple smart contract or maybe just use NEAR's wallet to send or account for tokens, the user will want to select the network--select a specific blockchain, be it local or a live multi-user chain--to interact with. Today, all network configuration is contained in the ~/.near directory with only a single network configured at any given moment. The ~/.near directory could contain configurations for multiple networks. The network our documentation assumes today is testnet, and the ~/.near directory is configured whenever near-shell is started and the user attempts to create or log in to an account. + +This NEP proposes that NEAR Shell be enhanced to enable multiple networks to be added, configured, and removed using new commands under a network category of commands. + +Example: + +1. A user or developer wants to interact with a local development network to test a simple smart contract. +2. User installs NEAR Shell. +3. User runs NEAR Shell for the first time. If there is no network configured, NEAR Shell informs the user that a default network is configured for local interaction with a name of localnet. + +1. A user or developer wants to interact with NEARs active testnet network. +2. The user lists the networks currently configured: `near network list`. +3. The user, by default, is presented with a list of networks with only one entry: `localnet` +4. Since, after these enhancements, NEAR Shell creates a localnet by default, the user will need to add and select a new network configuration. The user executes the shell command: `near network add testnet ` + +etc. TBD. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +This is the technical portion of the NEP. Explain the design in sufficient detail that: + +- Its interaction with other features is clear. +- It is reasonably clear how the feature would be implemented. +- Corner cases are dissected by example. + +The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +* The download size of NEAR Shell will increase. The amount of increase will be determined by the code and libraries required to add the ability to deploy a validator node. +* The complexity of NEAR Shell increases whenever a new command is added. +* Today, the configuration of the NEAR networks is instigated and controlled using the `nearcore` repository. +* Not all users will want to run a validator node. +* These proposals increases the dependency chain of NEAR Shell to include Rust nearcore libraries and configuration. + + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +## Why is this design the best in the space of possible designs? + +Today is difficult to communicate and introduce distributed Web concepts and features to users and developers. To address this, NEAR developers and products should reduce the overall complexity of distributed Web application development and create a centralized interface to all NEAR distributed application features and functionality. + +Current best design practices for blockchain, wallets, and distributed Web applications attempt to mitigate blockchain interaction complexities by performing all required steps for interaction and feature use by *delaying* complex interactions and *encapsulating* multiple steps into as few interactions as possible. The enhancements proposed here embrace multiple facets of NEAR features by including the most complicated feature interactions (i.e., configuring multiple networks, account creation and configuration, starting and managing NEAR validator nodes, etc.) and allow complex interactions with NEAR's blockchain features to be delayed until absolutely required, i.e., validator node configuration and deployment is available immediately, on-demand, only as needed, with no extra steps required to run a local network or an active distributed blockchain network. The user is not required to pull a separate repository, hence *delaying* a relatively complex product interaction and practically eliminating most of the confusion introduced by fetching and inspecting another NEAR github repository. + +These NEAR Shell enhancements *encapsulate* multiple features and processes required to deploy and configure multiple networks. + + +- What other designs have been considered and what is the rationale for not choosing them? + +The current NEAR Shell design is the default. Other designs might be ones that do not include the ability to configure and deploy active validator nodes. Also, another approach might be to not use a shell at all. Instead, NEAR might create GUI applications, e.g., electron or Web server-based configuration management where the user starts a local application which provides a port towards which the user can point their local Web browser to configure NEAR accounts, networks, and nodes. + +- What is the impact of not doing this? + +Users will not have a single application which enables configuration and management of all NEAR features. Instead, multiple repositories with disparate commands and configuration will be required to use NEAR Protocol's distributed Web blockchain. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to resolve through the NEP process before this gets merged? + +NEAR developers must weigh in on the impact of including validator node configuration. Also, the `~/.near` directory is currently created by `nearcore`. Decisions must be made about whether or not NEAR's configuration should be managed by NEAR Shell instead of whenever a validator node is executed. + +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? + +The management of the `~/.near` directory should be resolved when these enhancements are stabilized. Also, the selection of the categories of commands and the commands themselves will need to be fleshed out to optimize both user and developer experiences when interacting with NEAR Protocol's distributed Web blockchain platform offering. + +- What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP? + +Out of scope is smart contract development and debugging. However, future versions of NEAR Shell might include specific commands that enable the debugging and optimization of smart contract deployments. + + +# Future possibilities +[future-possibilities]: #future-possibilities + +Think about what the natural extension and evolution of your proposal would +be and how it would affect the project as a whole in a holistic +way. Try to use this section as a tool to more fully consider all possible +interactions with the project in your proposal. +Also consider how the this all fits into the roadmap for the project +and of the relevant sub-team. + +This is also a good place to "dump ideas", if they are out of scope for the +NEP you are writing but otherwise related. + +If you have tried and cannot think of any future possibilities, +you may simply state that you cannot think of anything. + +Note that having something written down in the future-possibilities section +is not a reason to accept the current or a future NEP. Such notes should be +in the section on motivation or rationale in this or subsequent NEPs. +The section merely provides additional information. From 2b09d2988c43680f764eae329b18eb46b67a5213 Mon Sep 17 00:00:00 2001 From: Mike Purvis Date: Thu, 27 Feb 2020 19:56:17 -0800 Subject: [PATCH 02/13] flesh out translation, location of key and config storage, massage commands --- text/0000-near-shell.md | 152 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 137 insertions(+), 15 deletions(-) diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md index fec87e138..c9fca94f9 100644 --- a/text/0000-near-shell.md +++ b/text/0000-near-shell.md @@ -19,15 +19,114 @@ These enhancements are intended to simplify and enhance developer and user produ # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -Using NEAR Shell, users can configure accounts, networks, and an instance of a server validator node on the host machine the shell is running. An instance of NEAR Shell loads its state from the configuration directory and files stored in the default location: `~/.near`. Once the configuration is loaded, the active account and network are selected automatically either from the previously stored configuration or the defaults as shipped with NEAR Shell. +## Configuration and key management + +Configuration and key management are two types of stored settings necessary for developers to build and deploy projects. The options for key storage and the way configuration is loaded will benefit with customization. + +Definitions: +- Configuration - a key-value storage (typically JSON) containing information. + - `networkId` - similar to an environment to work on (ex: 'staging') + - `nodeUrl` - URL to RPC + - `contractName` - NEAR account name for smart contract + - `walletUrl` - URL to NEAR Wallet + - `helperUrl` - URL to project helping with token dissemination + - `masterAccount` - NEAR account used for continuous integration + - `keyPath` - for validators, the path to a file containing an account's private key +- Keys - the storage of an account id and a corresponding private key. + - `account_id` + - `private_key` + +### Projects (currently) + +Currently, NEAR projects (for example, one generated using `create-near-app`) hold the configuration in the project's file: + +`src/config.js` + +**Note**: the active configuration can have keys overridden by flags. + +After the user completes the instructions from running `near login`, the project has a new directory where private key(s) are stored: + +`./neardev` + +### Validator nodes (currently) + +Currently, validators use different scripts + +Validator nodes using the `nearcore` repository follow instructions that will create their keys in the user's home directory, not the project directory. This directory name differs from a project's key storage directory. It is located in: + +`~/.near` + +### Proposed customization + +Both configuration and key management will be able to exist in various places, allowing end users to customize which accounts are using which configuration. Keys and config settings will apply for all contracts unless explicitly directed via the usage of CLI flags. + +Commands check the active configuration for items not found, or expected in flags. + +For example, instead of a user entering: + +`near call my_dapp my_function --accountId my_account_id` + +there exist an configuration that substitutes the `--accountId` flag: + +```json +… + "accountId": "my_account_id", +… +``` + +`near-shell` will look for keys in a specific order. If the active configuration file explicitly specifies a key storage directive, that setting will take precedence and it will not use this lookup order. This list is in ascending order and can be understood to mean, "if the key is not found here, then try the next location/store." + +1. Environment variables: + - `NEAR_ACCOUNT_ID` + - `NEAR_PRIVATE_KEY` + +2. Operating system key management: + - OS X - use of built-in `/usr/bin/security` [cli tool](https://www.unix.com/man-page/osx/1/security/). + - Linux - use of `secret-tool` [cli command](https://specifications.freedesktop.org/secret-service/latest/). + - Windows - possibly [use cmdkey.exe](https://social.technet.microsoft.com/Forums/en-US/268cb72e-0916-4219-8543-219092d2fb39/command-line-for-credential-manager?forum=w7itprosecurity) although implementation is not certain + +3. Project directory (`/Users/friend/projects/my-awesome-app/.near`) + - Instead of `neardev` in the project directory, it is now called `.near` + +4. Home directory (`/Users/friend/.near`) + +**Note**: during implementation it's advised to have the key storage options extendable. As the project grows, developers in the NEAR Collective may choose to add integrations with password management applications or hosted key solutions. + +## Translation + +It's important to invite the international community into developing with NEAR. Translation of these user-facing content are essential: +- Help commands (the text describing what a command does and how to use it) +- Error and logging messages + +Language preference can be set in two ways: +- Using environment variables (i.e., parsed from `process.env.LANG`) +- Set explicitly in configuration using [ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) + +```json +… + "defaultAccountId": "my_account_id", + "language": "en" +… +``` + +Commands and subcommands (ex: `call`) are not translated. + +#### User stories +As a user, I want to store my keys safely in my home directory, but have a configuration file in the project folder that overrides my default configuration stored in the home directory. + +As a user, I want the operating system to prompt for my user password deploying a contract. + +As a user, I want to be able to have multiple projects that use the same account (and corresponding key(s)) without having to login to each project. + +As a user, I want to be able to read logs and instructions for commands in the language locale I'm used to, or I choose. ## NEAR Shell Top-Level Commands ### `near network ` -This command category is used to select and configure NEAR networks for a NEAR Shell user. This category manipulates the user's `~/.near` configuration to allow a user to specify NEAR network details via the CLI instead of manually editing config files. +This command category is used to select and configure NEAR networks for a NEAR Shell user. This category manipulates the user's active configuration to allow a user to specify NEAR network details via the CLI instead of manually editing config files. -Sub-commands for `network` might include: +Sub-commands for `network` include: * `near network list` : Display the current list of networks for the current instance of NEAR Shell. * `near network status` : Show the user's current network configuration. @@ -42,12 +141,15 @@ Sub-commands for `network` might include: This category is used to create, select, and configure accounts on NEAR networks for a given NEAR Shell user. * `near account list` : Display list of accounts configured on local host. -* `near account status` : Display status of active or specified account. +* `near account status` : Display status of active or specified account, including token amounts, locked, etc. +* `near account send` : Send tokens from active or specified account to another NEAR account or contract. * `near account create` : Create a new account or sub-account. * `near account delete` : Delete an account or sub-account. +* `near account create-key` : Delete an account or sub-account. +* `near account revoke-key` : Delete an account or sub-account. * `near account select` : Select an account from the list of locally configured accounts as the active account. * `near account login` : Log in the current active or specified account. -* `near account logout` : Log out the current active or specified account. +* `near account logout` : Log out the current active or specified account. Essentially revoking a full access key. * `near account help` : Display help on account subcommands. ### `server` @@ -60,15 +162,7 @@ These commands are used to administer a NEAR server validator node. Currently, * `near server monitor` : Interactively display NEAR validator server status on current host. * `near server tail` : Tail the log of the NEAR validator server on current host. * `near server help` : Display help on server subcommands. - -### `near tokens ` - -* `near tokens send` : Send tokens to another NEAR account. -* `near tokens status` : Display tokens in wallet. -* `near tokens stake` -* `near tokens help` - -### `near key ` +* `near server stake` : Stake tokens on the configured network with the active or specified account ### `near contract ` * `near contract list` @@ -78,9 +172,23 @@ These commands are used to administer a NEAR server validator node. Currently, * `near contract build` * `near contract deploy` * `near contract call` +* `near contract view` ### `near config ` +* `near config` : List the location of active configuration file if it's loaded, or default config settings +* `near config set` : Set a key in the active configuration, or if using default, create a config file in the home directory with defaults and the specified key and value + +## Command input can be inline and file-based + +Some commands in `near-shell` may become long and difficult to type out. End users with a standard Terminal application may have a difficult time making small adjustments to a command that is quite lengthy. (Ex: pressing backwards or forwards many times to correct a typo.) + +In this spec, flags may be done inline: +`…call my_dapp my_function -i '{"key": "value_19"}'` + +or designated by a file: + +`…call my_dapp my_function -f ./params.json` ### User Stories For user-facing NEPs this section should focus on user stories. @@ -91,6 +199,8 @@ If NEAR Shell is required, it makes sense to enhance NEAR Shell to include other Here are a few user stories to demonstrate the utility of these enhancements. +1. As a user, I want to + #### Creating and Configuring Networks When a user or developer wants to try out a simple smart contract or maybe just use NEAR's wallet to send or account for tokens, the user will want to select the network--select a specific blockchain, be it local or a live multi-user chain--to interact with. Today, all network configuration is contained in the ~/.near directory with only a single network configured at any given moment. The ~/.near directory could contain configurations for multiple networks. The network our documentation assumes today is testnet, and the ~/.near directory is configured whenever near-shell is started and the user attempts to create or log in to an account. @@ -106,7 +216,7 @@ Example: 1. A user or developer wants to interact with NEARs active testnet network. 2. The user lists the networks currently configured: `near network list`. 3. The user, by default, is presented with a list of networks with only one entry: `localnet` -4. Since, after these enhancements, NEAR Shell creates a localnet by default, the user will need to add and select a new network configuration. The user executes the shell command: `near network add testnet ` +4. Since, after these enhancements, near-shell creates a localnet by default, the user will need to add and select a new network configuration. The user executes the shell command: `near network add testnet ` etc. TBD. @@ -121,6 +231,18 @@ This is the technical portion of the NEP. Explain the design in sufficient detai The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. +--- + +The programming language for NEAR Shell has not been determined yet but there have been opinions regarding proper, future-proof implementation. + +Long term goals to keep in mind: +- Upgradable, but also able to lock a specific version. +- Where possible, avoid single point of failures. This is in regards to using a package manager in particular. + - An official installer for multiple operating systems may be advised here, where trusted OS or GPG keys can verify the shell. +- Able to traverse directory structures from all operating systems + - This applies particularly to Windows where backslashes are used in the default command prompt. +- Limited dependencies to abate unforeseen issues. + # Drawbacks [drawbacks]: #drawbacks From fc863d45bef956f5939d373afe9a743d1c87bc8d Mon Sep 17 00:00:00 2001 From: Mike Purvis Date: Tue, 14 Apr 2020 17:01:43 -0700 Subject: [PATCH 03/13] add upgradability, project-level settings notion --- text/0000-near-shell.md | 245 +++++++++++++++++++++++++++------------- 1 file changed, 168 insertions(+), 77 deletions(-) diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md index c9fca94f9..0cfcb35ef 100644 --- a/text/0000-near-shell.md +++ b/text/0000-near-shell.md @@ -6,43 +6,25 @@ # Summary [summary]: #summary -NEAR Shell is one of the primary tools NEAR developers must use to develop and interact with NEAR networks and smart contracts. Today, NEAR Shell contains the base commands for account creation and deletion, login, viewing account state and keys, sending tokens, creating staking transactions, and building, deploying, and calling smart contracts. There are more opportunities to integrate core NEAR features to expose them directly to users. This proposal is to enhance NEAR Shell to represent all major functionality of NEAR. For example, validator node management, network selection, native OS key management and integration, smart contract development (including testing, and interaction) could be added. +`near-shell` is one of the primary tools NEAR developers must use to develop and interact with NEAR networks and smart contracts. Today, `near-shell` contains the base commands for account creation and deletion, login, viewing account state and keys, sending tokens, creating staking transactions, and building, deploying, and calling smart contracts. There are more opportunities to integrate core NEAR features to expose them directly to users. This proposal is to enhance `near-shell` to represent all major functionality of NEAR. For example, validator node management, network selection, native OS key management and integration, smart contract development (including testing, and interaction) could be added. -NEAR Shell is an excellent tool to attract and engage new NEAR developers, both core and contract level, and if it contains all the required features to select and configure and deploy NEAR networks, nodes, crypto assets (i.e., keys), accounts, and wallets, it can be the ideal single-entry point for those potential new hackers. +`near-shell` is an excellent tool to attract and engage new NEAR developers, both core and contract level, and if it contains all the required features to select and configure and deploy NEAR networks, nodes, crypto assets (i.e., keys), accounts, and wallets, it can be the ideal single-entry point for those potential new hackers. # Motivation [motivation]: #motivation -These enhancements are intended to simplify and enhance developer and user productivity by providing a single command-line tool which exercises all major features of NEAR. As each major piece of NEAR functionality is added to Near Shell, we should see utilization of each of those features increase. Additionally, community mindshare and understanding of NEAR should increase as NEAR Shell will contain detailed help documentation for each major feature with clear examples of use, just like Linux man pages and/or other major CLI tools which have integrated help for each command. - +These enhancements are intended to simplify and enhance developer and user productivity by providing a single command-line tool which exercises all major features of NEAR. As each major piece of NEAR functionality is added to `near-shell`, we should see utilization of each of those features increase. Additionally, community mindshare and understanding of NEAR should increase as `near-shell` will contain detailed help documentation for each major feature with clear examples of use, just like Linux man pages and/or other major CLI tools which have integrated help for each command. # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -## Configuration and key management - -Configuration and key management are two types of stored settings necessary for developers to build and deploy projects. The options for key storage and the way configuration is loaded will benefit with customization. - -Definitions: -- Configuration - a key-value storage (typically JSON) containing information. - - `networkId` - similar to an environment to work on (ex: 'staging') - - `nodeUrl` - URL to RPC - - `contractName` - NEAR account name for smart contract - - `walletUrl` - URL to NEAR Wallet - - `helperUrl` - URL to project helping with token dissemination - - `masterAccount` - NEAR account used for continuous integration - - `keyPath` - for validators, the path to a file containing an account's private key -- Keys - the storage of an account id and a corresponding private key. - - `account_id` - - `private_key` - ### Projects (currently) Currently, NEAR projects (for example, one generated using `create-near-app`) hold the configuration in the project's file: `src/config.js` -**Note**: the active configuration can have keys overridden by flags. +**Note**: the active configuration can have values overridden by flags. After the user completes the instructions from running `near login`, the project has a new directory where private key(s) are stored: @@ -50,31 +32,51 @@ After the user completes the instructions from running `near login`, the project ### Validator nodes (currently) -Currently, validators use different scripts +Currently, validators use scripts to run a node. -Validator nodes using the `nearcore` repository follow instructions that will create their keys in the user's home directory, not the project directory. This directory name differs from a project's key storage directory. It is located in: +Validator nodes using the `nearcore` repository follow instructions that will create their keys in the user's home directory, not the project directory. It is located in: `~/.near` -### Proposed customization +At this time, it is not advised to change this directory or try to use it for multiple purposes. Validator data has a monopoly on that folder. -Both configuration and key management will be able to exist in various places, allowing end users to customize which accounts are using which configuration. Keys and config settings will apply for all contracts unless explicitly directed via the usage of CLI flags. +## Settings, config, and key management -Commands check the active configuration for items not found, or expected in flags. +()From this section forward, the discussion is no longer about how `near-shell` works at the time of this writing, but what is proposed.) -For example, instead of a user entering: +Project-level **settings**, connection **configuration**, and **key management** are the three types of stored data necessary for developers to reliably build and deploy projects. -`near call my_dapp my_function --accountId my_account_id` +### Definitions: -there exist an configuration that substitutes the `--accountId` flag: +#### Settings +[project-level-settings]: #project-level-settings +These are key-values reflecting how `near-shell` behaves when called within a NEAR project. + +**Example**: a developer using OS X uses `near login` with access to a browser, whereas a validator runs `near login` on a CentOS box with no UI or browser. A user prompt may ask "Is this login from a computer with a browser?" The answer is then stored in key-value format in the project-level directory so it can be referenced later, skipping the prompt in the future. -```json -… - "accountId": "my_account_id", -… -``` +Besides answers to user prompts, project-level settings also store information about the last version of `near-shell` used in this project. For more information, please see [Upgradability](#upgradability). + +#### Configuration +A key-value storage (typically JSON) containing information. + +- `networkId` - similar to an environment to work on (ex: 'staging') +- `nodeUrl` - URL to RPC +- `contractName` - NEAR account name for smart contract +- `walletUrl` - URL to NEAR Wallet +- `helperUrl` - URL to project helping with token dissemination +- `masterAccount` - NEAR account used for continuous integration + +**Example**: As a user, I want to deploy a contract to my localnet instead of testnet during development. I will provide flags and/or configuration so that `near-shell` can determine where to connect, for what contract, and on behalf of which NEAR account. -`near-shell` will look for keys in a specific order. If the active configuration file explicitly specifies a key storage directive, that setting will take precedence and it will not use this lookup order. This list is in ascending order and can be understood to mean, "if the key is not found here, then try the next location/store." +#### Key management +The storage of an account id and a corresponding private key. + - `keyPath` - an argument used by `near-shell` specifying the path to the unencrypted file containing an account's private key. + +The unencrypted file contains the keys: +- `account_id` +- `private_key` + +`near-shell` will look for keys in a specific order. This list is in the prioritized order and can be understood to mean, "if the key is not found here, then try the next location/store." 1. Environment variables: - `NEAR_ACCOUNT_ID` @@ -86,59 +88,135 @@ there exist an configuration that substitutes the `--accountId` flag: - Windows - possibly [use cmdkey.exe](https://social.technet.microsoft.com/Forums/en-US/268cb72e-0916-4219-8543-219092d2fb39/command-line-for-credential-manager?forum=w7itprosecurity) although implementation is not certain 3. Project directory (`/Users/friend/projects/my-awesome-app/.near`) - - Instead of `neardev` in the project directory, it is now called `.near` + - Instead of `neardev` in the project directory, it is now called `.near-credentials` -4. Home directory (`/Users/friend/.near`) +4. Home directory (`/Users/friend/.near-credentials`) **Note**: during implementation it's advised to have the key storage options extendable. As the project grows, developers in the NEAR Collective may choose to add integrations with password management applications or hosted key solutions. ## Translation -It's important to invite the international community into developing with NEAR. Translation of these user-facing content are essential: +It's important to invite the international community into developing with NEAR. Translation of user-facing content is essential for: - Help commands (the text describing what a command does and how to use it) - Error and logging messages Language preference can be set in two ways: -- Using environment variables (i.e., parsed from `process.env.LANG`) -- Set explicitly in configuration using [ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) +- Using environment variables (i.e., parsed from `process.env.LANG`, and the default assignment) +- Set explicitly in [project-level settings](#project-level-settings) using [ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) +Example content from `/.near-config/settings.env`: ```json … - "defaultAccountId": "my_account_id", "language": "en" … ``` -Commands and subcommands (ex: `call`) are not translated. +Commands and subcommands (ex: `call` or `deploy`) are not translated and will be in English. -#### User stories -As a user, I want to store my keys safely in my home directory, but have a configuration file in the project folder that overrides my default configuration stored in the home directory. +## Upgradability +[upgradability]: #upgradability -As a user, I want the operating system to prompt for my user password deploying a contract. +As `near-shell` matures, updates may/will cause a user's project to become outdated. For instance, this NEP proposes to rename the project directory `neardev` to `.near-credentials`. Developers with existing `neardev` directories will either need to manually rename the folder, or `near-shell` will have to look in two places for the key files. -As a user, I want to be able to have multiple projects that use the same account (and corresponding key(s)) without having to login to each project. +This proposal declares that `near-shell` will have a mechanism to make such upgrades possible. This is not possible, however, without keeping track of which version of `near-shell` was most recently used on a project. + +**Example scenario**: a user begins developing a dApp on NEAR using `near-shell` version 0.19.0. The user takes a sabbatical for a few months and returns to the project with a new computer. The new computer installed `near-shell` version 0.23.1. The first time this user runs a command in this old project, migrations are run ensuring the project stays current. + +Of the three types of storage mentioned before (project-level settings, connection configuration, and key management) the saved version information belongs to the project-level settings. It's possible that a user will have multiple dApps that are used less frequently than others. Each project, especially those used less frequently, must have their own record of which version of `near-shell` was most recently used. + +The location of these settings is: + +`./.near-config/settings.env` for a project. + +For example: +`/Users/friend/near-projects/guest-book/.near-config/settings.env` + +The settings will exist as normal key-value pairs seen in environment variable files: + +```bash +… +LAST_SHELL_VERSION="0.19.1" +… +``` + +Using middleware, `near-shell` will check the current version against the key `LAST_SHELL_VERSION` of the settings file and run any necessary migrations. -As a user, I want to be able to read logs and instructions for commands in the language locale I'm used to, or I choose. +### Migrations -## NEAR Shell Top-Level Commands +The folder structure within `near-shell` will have migration files per minor version as illustrated here: + +```bash +└── middleware + └── migration + ├── README.md + ├── scripts + │ ├── 0.19.x.js ⟵ logic to run for 0.19.0 to 0.19.N + │ ├── 0.20.x.js ⟵ logic to run for 0.20.0 to 0.20.N + │ └── x.x.x-template.js ⟵ template for adding new minor version migrations + └── shell-upgrade.js ⟵ contains checks for current version against stored version, etc. +``` + +An example of a migration script might be: + +```javascript +const upgrade = async (lastPatchVersion) => { + if (lastPatchVersion < 2) { + // implement essential logic that changed from x.x.0 to x.x.1 + // check for existence of neardev/dev-account.env file, create if possible + const fs = require('fs'); + const util = require('util'); + + (async () => { + const oldDevDeployFile = 'neardev/dev-account'; + const newDevDeployFile = 'neardev/dev-account.env'; + + if (fs.existsSync(newDevDeployFile)) { + // user already has the latest near-shell + return; + } + + if (fs.existsSync(oldDevDeployFile)) { + // user has an outdated near-shell, create necessary file + const readFile = (filePath) => util.promisify(fs.readFile)(filePath, 'utf8'); + const writeFile = (filePath, data) => util.promisify(fs.writeFile)(filePath, data); + // read and rewrite the old file into the new format + const fileData = await readFile(oldDevDeployFile); + await writeFile(newDevDeployFile, `CONTRACT_NAME=${fileData}`); + } + })(); + } + + if (lastPatchVersion < 6) { + // implement essential logic that changed from x.x.2 to x.x.6 + } + + … +}; + +exports.upgrade = upgrade; +``` + +The file `shell-upgrade.js`, after comparing the current version to the setting in `./.near-config/settings.env`, will determine how many migration scripts need to run, and call the `upgrade()` function on them in order. When complete, it will update the `LAST_SHELL_VERSION` key in the project-level settings file. At this time the project is considered current. + +## `near-shell` Top-Level Commands ### `near network ` -This command category is used to select and configure NEAR networks for a NEAR Shell user. This category manipulates the user's active configuration to allow a user to specify NEAR network details via the CLI instead of manually editing config files. +This command category is used to select and configure NEAR networks for a `near-shell` user. This category manipulates the user's active configuration to allow a user to specify NEAR network details via the CLI instead of manually editing config files. Sub-commands for `network` include: -* `near network list` : Display the current list of networks for the current instance of NEAR Shell. +* `near network list` : Display the current list of networks for the current instance of `near-shell`. * `near network status` : Show the user's current network configuration. -* `near network select` : Select a NEAR network as the default network for subsequent NEAR Shell commands as well as the network configuration for a local server validator node managed by this instance of NEAR Shell. +* `near network select` : Select a NEAR network as the default network for subsequent `near-shell` commands as well as the network configuration for a local server validator node managed by this instance of `near-shell`. * `near network add` : Add a network to the users's `~/.near` config files. -* `near network remove` : Remove a network from tthe `~/.near` config files. +* `near network remove` : Remove a network from the `~/.near` config files. * `near network monitor` : Interactively monitor one or more NEAR networks in a curses-like interface updated in a specified interval, default of 1s. * `near network help` : Display help on network subcommands. ### `near account ` -This category is used to create, select, and configure accounts on NEAR networks for a given NEAR Shell user. +This category is used to create, select, and configure accounts on NEAR networks for a given `near-shell` user. * `near account list` : Display list of accounts configured on local host. * `near account status` : Display status of active or specified account, including token amounts, locked, etc. @@ -154,7 +232,7 @@ This category is used to create, select, and configure accounts on NEAR networks ### `server` -These commands are used to administer a NEAR server validator node. Currently, only one server per NEAR Shell is proposed; however, multiple servers per physical host should be considered. +These commands are used to administer a NEAR server validator node. Currently, only one server per `near-shell` is proposed; however, multiple servers per physical host should be considered. * `near server status` : Display status of NEAR validator server on current host. * `near server start` : Start NEAR validator server on current host. @@ -180,43 +258,54 @@ These commands are used to administer a NEAR server validator node. Currently, ## Command input can be inline and file-based -Some commands in `near-shell` may become long and difficult to type out. End users with a standard Terminal application may have a difficult time making small adjustments to a command that is quite lengthy. (Ex: pressing backwards or forwards many times to correct a typo.) +Some commands in `near-shell` may become long and difficult to type on the command line. End users with a standard terminal application may have lengthy arguments that are better saved to a file. -In this spec, flags may be done inline: +Flags may be done inline (default): -`…call my_dapp my_function -i '{"key": "value_19"}'` +`near call my_dapp my_function '{"key": "value_19"}'` -or designated by a file: +or defined in a file: -`…call my_dapp my_function -f ./params.json` +`near call my_dapp my_function -f ./params.json` + +This `-f` or `--fromFile` argument is added to two commands: +1. `call` +2. `view` + +Other commands may be added in the future. ### User Stories -For user-facing NEPs this section should focus on user stories. -These enhancements to near-shell aim to increase user and developer adoption by centralizing and optimizing all interactions with NEAR functionality. Near Shell today is a thin and lightweight set of features which enable the use of each NEAR product offering. Specifically, Near Shell is required to use NEAR products today. For example, to stake, send tokens, create and delete accounts, view public keys, build and deploy smart contracts, call smart contract methods, and log in through NEAR Protocol's wallet. Each of these features are critical to all NEAR products. +These enhancements to `near-shell` aim to increase user and developer adoption by centralizing and optimizing all interactions with NEAR functionality. `near-shell` today is a thin and lightweight set of features which enable the use of each NEAR product offering. Specifically, `near-shell` is required to use NEAR products today. For example, to stake, send tokens, create and delete accounts, view public keys, build and deploy smart contracts, call smart contract methods, and log in through NEAR Protocol's wallet. Each of these features are critical to all NEAR products. -If NEAR Shell is required, it makes sense to enhance NEAR Shell to include other features critical to user and developer interactions. The commands proposed above enhance NEAR Shell to include validator node deployment and control for both local and distributed development, i.e., using a local network for offline development vs. testnet as a live network used by others, and to be able to select configure and select networks. +If `near-shell` is required, it makes sense to enhance `near-shell` to include other features critical to user and developer interactions. The commands proposed above enhance `near-shell` to include validator node deployment and control for both local and distributed development, i.e., using a local network for offline development vs. testnet as a live network used by others, and to be able to select configure and select networks. Here are a few user stories to demonstrate the utility of these enhancements. -1. As a user, I want to +As a user, I want the option to have my operating system prompt for my user password before deploying a contract. + +As a user, I want to be able to have multiple projects that use the same account (and corresponding key(s)) without having to login to each project. + +As a user, I want to be able to instructions for commands in my own language. + +As a user, I want to be able to come back to a project months later, upgrade tooling, and face no issues building and deploying. #### Creating and Configuring Networks -When a user or developer wants to try out a simple smart contract or maybe just use NEAR's wallet to send or account for tokens, the user will want to select the network--select a specific blockchain, be it local or a live multi-user chain--to interact with. Today, all network configuration is contained in the ~/.near directory with only a single network configured at any given moment. The ~/.near directory could contain configurations for multiple networks. The network our documentation assumes today is testnet, and the ~/.near directory is configured whenever near-shell is started and the user attempts to create or log in to an account. +When a user or developer wants to try out a simple smart contract or maybe just use NEAR's wallet to send or account for tokens, the user will want to select the network--select a specific blockchain, be it local or a live multi-user chain--to interact with. Today, all network configuration is contained in the ~/.near directory with only a single network configured at any given moment. The ~/.near directory could contain configurations for multiple networks. The network our documentation assumes today is testnet, and the ~/.near directory is configured whenever `near-shell` is started and the user attempts to create or log in to an account. -This NEP proposes that NEAR Shell be enhanced to enable multiple networks to be added, configured, and removed using new commands under a network category of commands. +This NEP proposes that `near-shell` be enhanced to enable multiple networks to be added, configured, and removed using new commands under a network category of commands. Example: 1. A user or developer wants to interact with a local development network to test a simple smart contract. -2. User installs NEAR Shell. -3. User runs NEAR Shell for the first time. If there is no network configured, NEAR Shell informs the user that a default network is configured for local interaction with a name of localnet. +2. User installs `near-shell`. +3. User runs `near-shell` for the first time. If there is no network configured, `near-shell` informs the user that a default network is configured for local interaction with a name of localnet. 1. A user or developer wants to interact with NEARs active testnet network. 2. The user lists the networks currently configured: `near network list`. 3. The user, by default, is presented with a list of networks with only one entry: `localnet` -4. Since, after these enhancements, near-shell creates a localnet by default, the user will need to add and select a new network configuration. The user executes the shell command: `near network add testnet ` +4. Since, after these enhancements, `near-shell` creates a localnet by default, the user will need to add and select a new network configuration. The user executes the shell command: `near network add testnet ` etc. TBD. @@ -233,7 +322,7 @@ The section should return to the examples given in the previous section, and exp --- -The programming language for NEAR Shell has not been determined yet but there have been opinions regarding proper, future-proof implementation. +The programming language for `near-shell` has not been determined yet but there have been opinions regarding proper, future-proof implementation. Long term goals to keep in mind: - Upgradable, but also able to lock a specific version. @@ -248,11 +337,11 @@ Long term goals to keep in mind: Why should we *not* do this? -* The download size of NEAR Shell will increase. The amount of increase will be determined by the code and libraries required to add the ability to deploy a validator node. -* The complexity of NEAR Shell increases whenever a new command is added. +* The download size of `near-shell` will increase. The amount of increase will be determined by the code and libraries required to add the ability to deploy a validator node. +* The complexity of `near-shell` increases whenever a new command is added. * Today, the configuration of the NEAR networks is instigated and controlled using the `nearcore` repository. * Not all users will want to run a validator node. -* These proposals increases the dependency chain of NEAR Shell to include Rust nearcore libraries and configuration. +* These proposals increases the dependency chain of `near-shell` to include Rust nearcore libraries and configuration. # Rationale and alternatives @@ -264,12 +353,12 @@ Today is difficult to communicate and introduce distributed Web concepts and fea Current best design practices for blockchain, wallets, and distributed Web applications attempt to mitigate blockchain interaction complexities by performing all required steps for interaction and feature use by *delaying* complex interactions and *encapsulating* multiple steps into as few interactions as possible. The enhancements proposed here embrace multiple facets of NEAR features by including the most complicated feature interactions (i.e., configuring multiple networks, account creation and configuration, starting and managing NEAR validator nodes, etc.) and allow complex interactions with NEAR's blockchain features to be delayed until absolutely required, i.e., validator node configuration and deployment is available immediately, on-demand, only as needed, with no extra steps required to run a local network or an active distributed blockchain network. The user is not required to pull a separate repository, hence *delaying* a relatively complex product interaction and practically eliminating most of the confusion introduced by fetching and inspecting another NEAR github repository. -These NEAR Shell enhancements *encapsulate* multiple features and processes required to deploy and configure multiple networks. +These `near-shell` enhancements *encapsulate* multiple features and processes required to deploy and configure multiple networks. - What other designs have been considered and what is the rationale for not choosing them? -The current NEAR Shell design is the default. Other designs might be ones that do not include the ability to configure and deploy active validator nodes. Also, another approach might be to not use a shell at all. Instead, NEAR might create GUI applications, e.g., electron or Web server-based configuration management where the user starts a local application which provides a port towards which the user can point their local Web browser to configure NEAR accounts, networks, and nodes. +The current `near-shell` design is the default. Other designs might be ones that do not include the ability to configure and deploy active validator nodes. Also, another approach might be to not use a shell at all. Instead, NEAR might create GUI applications, e.g., electron or Web server-based configuration management where the user starts a local application which provides a port towards which the user can point their local Web browser to configure NEAR accounts, networks, and nodes. - What is the impact of not doing this? @@ -280,7 +369,9 @@ Users will not have a single application which enables configuration and managem - What parts of the design do you expect to resolve through the NEP process before this gets merged? -NEAR developers must weigh in on the impact of including validator node configuration. Also, the `~/.near` directory is currently created by `nearcore`. Decisions must be made about whether or not NEAR's configuration should be managed by NEAR Shell instead of whenever a validator node is executed. +aloha - remove all the stuff about ~/.near + +NEAR developers must weigh in on the impact of including validator node configuration. Also, the `~/.near` directory is currently created by `nearcore`. Decisions must be made about whether or not NEAR's configuration should be managed by `near-shell` instead of whenever a validator node is executed. - What parts of the design do you expect to resolve through the implementation of this feature before stabilization? @@ -288,7 +379,7 @@ The management of the `~/.near` directory should be resolved when these enhancem - What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP? -Out of scope is smart contract development and debugging. However, future versions of NEAR Shell might include specific commands that enable the debugging and optimization of smart contract deployments. +Out of scope is smart contract development and debugging. However, future versions of `near-shell` might include specific commands that enable the debugging and optimization of smart contract deployments. # Future possibilities From f9ed4749c7b5fdab834fbb8e23f839eed0f94c03 Mon Sep 17 00:00:00 2001 From: Mike Purvis Date: Wed, 15 Apr 2020 11:05:23 -0700 Subject: [PATCH 04/13] cleanup, add prompt details, fix priority wording, move network to future --- text/0000-near-shell.md | 201 +++++++++++++++++++--------------------- 1 file changed, 93 insertions(+), 108 deletions(-) diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md index 0cfcb35ef..a02b3d2d1 100644 --- a/text/0000-near-shell.md +++ b/text/0000-near-shell.md @@ -18,32 +18,8 @@ These enhancements are intended to simplify and enhance developer and user produ # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -### Projects (currently) - -Currently, NEAR projects (for example, one generated using `create-near-app`) hold the configuration in the project's file: - -`src/config.js` - -**Note**: the active configuration can have values overridden by flags. - -After the user completes the instructions from running `near login`, the project has a new directory where private key(s) are stored: - -`./neardev` - -### Validator nodes (currently) - -Currently, validators use scripts to run a node. - -Validator nodes using the `nearcore` repository follow instructions that will create their keys in the user's home directory, not the project directory. It is located in: - -`~/.near` - -At this time, it is not advised to change this directory or try to use it for multiple purposes. Validator data has a monopoly on that folder. - ## Settings, config, and key management -()From this section forward, the discussion is no longer about how `near-shell` works at the time of this writing, but what is proposed.) - Project-level **settings**, connection **configuration**, and **key management** are the three types of stored data necessary for developers to reliably build and deploy projects. ### Definitions: @@ -55,44 +31,60 @@ These are key-values reflecting how `near-shell` behaves when called within a NE **Example**: a developer using OS X uses `near login` with access to a browser, whereas a validator runs `near login` on a CentOS box with no UI or browser. A user prompt may ask "Is this login from a computer with a browser?" The answer is then stored in key-value format in the project-level directory so it can be referenced later, skipping the prompt in the future. Besides answers to user prompts, project-level settings also store information about the last version of `near-shell` used in this project. For more information, please see [Upgradability](#upgradability). + +In summary, "settings" are key-value pairs that are set by `near-shell` and relate to the behavior of how it operates in a given project directory. They do not relate to the development or deployment of smart contracts. #### Configuration -A key-value storage (typically JSON) containing information. - -- `networkId` - similar to an environment to work on (ex: 'staging') -- `nodeUrl` - URL to RPC -- `contractName` - NEAR account name for smart contract -- `walletUrl` - URL to NEAR Wallet -- `helperUrl` - URL to project helping with token dissemination -- `masterAccount` - NEAR account used for continuous integration +Configuration are key-value pairs that relate to development and deployment of smart contracts and refer to *connection* information. + +* `networkId` : Similar to an environment to work on (ex: 'staging') +* `nodeUrl` : URL to the NEAR node +* `contractName` : NEAR account name for the smart contract +* `walletUrl` : URL to NEAR Wallet +* `helperUrl` : URL to the contract helper that provides tokens upon account creation +* `masterAccount` : When creating "child" accounts, `masterAccount` is the "parent" +* `configFile` : Specifies the location of a config file containing connection information. Default location is `src/config.js` -**Example**: As a user, I want to deploy a contract to my localnet instead of testnet during development. I will provide flags and/or configuration so that `near-shell` can determine where to connect, for what contract, and on behalf of which NEAR account. +**Example**: As a user, I want to deploy a contract to my localnet instead of testnet during development. I will provide flags and/or modify configuration so that `near-shell` can determine where to connect, for what contract, and on behalf of which NEAR account. #### Key management The storage of an account id and a corresponding private key. - - `keyPath` - an argument used by `near-shell` specifying the path to the unencrypted file containing an account's private key. +* `keyPath` : An argument used by `near-shell` specifying the path to the keyfile. -The unencrypted file contains the keys: -- `account_id` -- `private_key` +The keyfile file contains the keys: +* `type` : Options include + - `unencrypted` : An unencrypted file containing an account's private key + - `native_osx` : Private key stored with OS X key management + - `native_linux` : Private key stored with Linux-based system's key management + - `native_windows` : Private key stored with Windows key management +* `account_id` : The NEAR account name +* `private_key` : The plain-text private key, used when `type = "unencrypted"` `near-shell` will look for keys in a specific order. This list is in the prioritized order and can be understood to mean, "if the key is not found here, then try the next location/store." 1. Environment variables: - `NEAR_ACCOUNT_ID` + - `NEAR_ACCOUNT_TYPE` - `NEAR_PRIVATE_KEY` -2. Operating system key management: - - OS X - use of built-in `/usr/bin/security` [cli tool](https://www.unix.com/man-page/osx/1/security/). - - Linux - use of `secret-tool` [cli command](https://specifications.freedesktop.org/secret-service/latest/). - - Windows - possibly [use cmdkey.exe](https://social.technet.microsoft.com/Forums/en-US/268cb72e-0916-4219-8543-219092d2fb39/command-line-for-credential-manager?forum=w7itprosecurity) although implementation is not certain -3. Project directory (`/Users/friend/projects/my-awesome-app/.near`) - - Instead of `neardev` in the project directory, it is now called `.near-credentials` +2. Project directory (`/Users/friend/projects/my-awesome-app/.near-credentials`) + + **Note**: Formerly, the `neardev` folder contained the key files for a project. It is now `.near-credentials`. + +3. Home directory (`/Users/friend/.near-credentials`) -4. Home directory (`/Users/friend/.near-credentials`) +If the key type among the `native_*` values, the operating system key management handles: -**Note**: during implementation it's advised to have the key storage options extendable. As the project grows, developers in the NEAR Collective may choose to add integrations with password management applications or hosted key solutions. +* OS X : Use of built-in `/usr/bin/security` [cli tool](https://www.unix.com/man-page/osx/1/security/). +* Linux : Use of `secret-tool` [cli command](https://specifications.freedesktop.org/secret-service/latest/). +* Windows : Possibly [use cmdkey.exe](https://social.technet.microsoft.com/Forums/en-US/268cb72e-0916-4219-8543-219092d2fb39/command-line-for-credential-manager?forum=w7itprosecurity) although implementation is not certain. + +This prioritized order allows project-level configuration to take priority over the home directory, offering an improved user experience. + +As an example, a user having credentials saved in their home directory will be able to use the `--accountId` flag to use keys from any project regardless of location. Said another way, the user does not need to run `near login` inside each project in order to access the keys. + +**Note**: as the project grows, developers in the NEAR Collective may choose to add integrations with password management applications or hosted key solutions. ## Translation @@ -120,18 +112,18 @@ As `near-shell` matures, updates may/will cause a user's project to become outda This proposal declares that `near-shell` will have a mechanism to make such upgrades possible. This is not possible, however, without keeping track of which version of `near-shell` was most recently used on a project. -**Example scenario**: a user begins developing a dApp on NEAR using `near-shell` version 0.19.0. The user takes a sabbatical for a few months and returns to the project with a new computer. The new computer installed `near-shell` version 0.23.1. The first time this user runs a command in this old project, migrations are run ensuring the project stays current. +**Example scenario**: a user begins developing a dApp on NEAR using `near-shell` version 0.19.0. The user takes a sabbatical for a few months and returns with a new computer, cloning the old project. The new computer installed `near-shell` version 0.23.1. The first time this user runs a command in this old project, migrations are run ensuring the project stays current. -Of the three types of storage mentioned before (project-level settings, connection configuration, and key management) the saved version information belongs to the project-level settings. It's possible that a user will have multiple dApps that are used less frequently than others. Each project, especially those used less frequently, must have their own record of which version of `near-shell` was most recently used. +Of the three types of storage mentioned before (project-level settings, connection configuration, and key management) the saved version information belongs to the project-level settings. It's possible that a user will have multiple dApps that are used less frequently than others. Each project must have their own record of which version of `near-shell` was most recently used. The location of these settings is: -`./.near-config/settings.env` for a project. +`.near-config/settings.js` for a project. For example: -`/Users/friend/near-projects/guest-book/.near-config/settings.env` +`/Users/friend/near-projects/guest-book/.near-config/settings.js` -The settings will exist as normal key-value pairs seen in environment variable files: +In this proposal, the file is shown as JavaScript. This is not a hard requirement, as as key-value pairs could also be in the form of environment variables: ```bash … @@ -139,7 +131,7 @@ LAST_SHELL_VERSION="0.19.1" … ``` -Using middleware, `near-shell` will check the current version against the key `LAST_SHELL_VERSION` of the settings file and run any necessary migrations. +Using middleware, `near-shell` will check the current version against the key `lastShellVersion` (or `LAST_SHELL_VERSION`) of the settings file, then run any necessary migrations. ### Migrations @@ -162,32 +154,14 @@ An example of a migration script might be: const upgrade = async (lastPatchVersion) => { if (lastPatchVersion < 2) { // implement essential logic that changed from x.x.0 to x.x.1 - // check for existence of neardev/dev-account.env file, create if possible - const fs = require('fs'); - const util = require('util'); - - (async () => { - const oldDevDeployFile = 'neardev/dev-account'; - const newDevDeployFile = 'neardev/dev-account.env'; - - if (fs.existsSync(newDevDeployFile)) { - // user already has the latest near-shell - return; - } - - if (fs.existsSync(oldDevDeployFile)) { - // user has an outdated near-shell, create necessary file - const readFile = (filePath) => util.promisify(fs.readFile)(filePath, 'utf8'); - const writeFile = (filePath, data) => util.promisify(fs.writeFile)(filePath, data); - // read and rewrite the old file into the new format - const fileData = await readFile(oldDevDeployFile); - await writeFile(newDevDeployFile, `CONTRACT_NAME=${fileData}`); - } - })(); + // Example: the new minor version 0.24.0 changes the neardev directory to .near-credentials + // Logic here that checks for the absense of .near-credentials, the existence of neardev, and renames accordingly } if (lastPatchVersion < 6) { // implement essential logic that changed from x.x.2 to x.x.6 + // Example: all keys in the .near-credentials need an additional key for "type" + // Logic looping through adding new key to existing files } … @@ -196,29 +170,37 @@ const upgrade = async (lastPatchVersion) => { exports.upgrade = upgrade; ``` -The file `shell-upgrade.js`, after comparing the current version to the setting in `./.near-config/settings.env`, will determine how many migration scripts need to run, and call the `upgrade()` function on them in order. When complete, it will update the `LAST_SHELL_VERSION` key in the project-level settings file. At this time the project is considered current. +Shown in the directory structure above is the file `shell-upgrade.js`. This file will, after comparing the current version to the last used version in `./.near-config/settings.js`, determine how many migration scripts are needed to run. It will loop through the necessary files calling the `upgrade()` function on them in the proper version order. When complete, it will update the `lastShellVersion` key in the project-level settings file. At this time the project is considered current. -## `near-shell` Top-Level Commands +**Note**: migrations do not have to fix backwards-incompatible changes. Migrations can also improve experience by, for example, removing orphaned files, make safety checks and show warnings, etc. -### `near network ` +## Prompts -This command category is used to select and configure NEAR networks for a `near-shell` user. This category manipulates the user's active configuration to allow a user to specify NEAR network details via the CLI instead of manually editing config files. +Various user prompts will enhance the experience of `near-shell` by providing options. These prompts have the option to save answers to that identical future prompts may be skipped if desired. As mentioned, the answers will be saved as **settings** on the project level. -Sub-commands for `network` include: +Example: +`/Users/friend/near-projects/.near-config/settings.js` has: -* `near network list` : Display the current list of networks for the current instance of `near-shell`. -* `near network status` : Show the user's current network configuration. -* `near network select` : Select a NEAR network as the default network for subsequent `near-shell` commands as well as the network configuration for a local server validator node managed by this instance of `near-shell`. -* `near network add` : Add a network to the users's `~/.near` config files. -* `near network remove` : Remove a network from the `~/.near` config files. -* `near network monitor` : Interactively monitor one or more NEAR networks in a curses-like interface updated in a specified interval, default of 1s. -* `near network help` : Display help on network subcommands. +```javascript +… +"alwaysSaveToLocation": "home" ⟵ example: user chose to always save keys to home directory when running "near login" +… +``` + +The number of prompts will grow beyond what can be captured in this spec. A number of possible prompts would be: + +- Is this project on an OS expected to have a browser and UI +- When creating an account, always fund new accounts with the NEAR contract helper +- Always run migrations in this project when applicable +- Never run migrations in this project + +## `near-shell` Commands ### `near account ` This category is used to create, select, and configure accounts on NEAR networks for a given `near-shell` user. -* `near account list` : Display list of accounts configured on local host. +* `near account list` : Display list of accounts configured in the current directory and home directory * `near account status` : Display status of active or specified account, including token amounts, locked, etc. * `near account send` : Send tokens from active or specified account to another NEAR account or contract. * `near account create` : Create a new account or sub-account. @@ -228,33 +210,20 @@ This category is used to create, select, and configure accounts on NEAR networks * `near account select` : Select an account from the list of locally configured accounts as the active account. * `near account login` : Log in the current active or specified account. * `near account logout` : Log out the current active or specified account. Essentially revoking a full access key. -* `near account help` : Display help on account subcommands. +* `near account --help` : Display help on account subcommands. -### `server` -These commands are used to administer a NEAR server validator node. Currently, only one server per `near-shell` is proposed; however, multiple servers per physical host should be considered. - -* `near server status` : Display status of NEAR validator server on current host. -* `near server start` : Start NEAR validator server on current host. -* `near server stop` : Stop NEAR validator server on current host. -* `near server monitor` : Interactively display NEAR validator server status on current host. -* `near server tail` : Tail the log of the NEAR validator server on current host. -* `near server help` : Display help on server subcommands. -* `near server stake` : Stake tokens on the configured network with the active or specified account +### `near config ` +* `near config` : List the location of active configuration file if it's loaded, or default config settings +* `near config set` : Set a key in the active configuration, or if using default, create a config file in the home directory with defaults and the specified key and value +* `near config clear` : Remove the [project-level settings](#project-level-settings) file ### `near contract ` -* `near contract list` -* `near contract status` -* `near contract add` -* `near contract remove` * `near contract build` * `near contract deploy` * `near contract call` * `near contract view` - -### `near config ` -* `near config` : List the location of active configuration file if it's loaded, or default config settings -* `near config set` : Set a key in the active configuration, or if using default, create a config file in the home directory with defaults and the specified key and value +* `near contract --help` ## Command input can be inline and file-based @@ -269,8 +238,8 @@ or defined in a file: `near call my_dapp my_function -f ./params.json` This `-f` or `--fromFile` argument is added to two commands: -1. `call` -2. `view` +1. `contract call` +2. `contract view` Other commands may be added in the future. @@ -312,6 +281,8 @@ etc. TBD. # Reference-level explanation [reference-level-explanation]: #reference-level-explanation +TODO: - move some stuff down here + This is the technical portion of the NEP. Explain the design in sufficient detail that: - Its interaction with other features is clear. @@ -369,7 +340,7 @@ Users will not have a single application which enables configuration and managem - What parts of the design do you expect to resolve through the NEP process before this gets merged? -aloha - remove all the stuff about ~/.near +TODO: - remove all the stuff about ~/.near NEAR developers must weigh in on the impact of including validator node configuration. Also, the `~/.near` directory is currently created by `nearcore`. Decisions must be made about whether or not NEAR's configuration should be managed by `near-shell` instead of whenever a validator node is executed. @@ -385,6 +356,20 @@ Out of scope is smart contract development and debugging. However, future versi # Future possibilities [future-possibilities]: #future-possibilities +### `near network ` + +This command category is used to select and configure NEAR networks for a `near-shell` user. This category manipulates the project's configuration to allow a user to specify NEAR network details via the CLI instead of manually editing config files. + +Sub-commands for `network` include: + +* `near network list` : Display the current list of networks for the current instance of `near-shell`. +* `near network status` : Show the user's current network configuration. +* `near network select` : Select a NEAR network as the default network for subsequent `near-shell` commands as well as the network configuration for a local server validator node managed by this instance of `near-shell`. +* `near network add` : Add a network to the users's `~/.near` config files. +* `near network remove` : Remove a network from the `~/.near` config files. +* `near network monitor` : Interactively monitor one or more NEAR networks in a curses-like interface updated in a specified interval, default of 1s. +* `near network help` : Display help on network subcommands. + Think about what the natural extension and evolution of your proposal would be and how it would affect the project as a whole in a holistic way. Try to use this section as a tool to more fully consider all possible From ef946bd7a97ae9836377c9d8146146e4d01376ac Mon Sep 17 00:00:00 2001 From: Mike Purvis Date: Wed, 15 Apr 2020 14:27:06 -0700 Subject: [PATCH 05/13] rework location of connection configuration, clarity across the board --- text/0000-near-shell.md | 102 ++++++++++++++++++++++++++++++---------- 1 file changed, 76 insertions(+), 26 deletions(-) diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md index a02b3d2d1..332019193 100644 --- a/text/0000-near-shell.md +++ b/text/0000-near-shell.md @@ -20,7 +20,19 @@ These enhancements are intended to simplify and enhance developer and user produ ## Settings, config, and key management -Project-level **settings**, connection **configuration**, and **key management** are the three types of stored data necessary for developers to reliably build and deploy projects. +Project-level **settings**, connection **configuration**, and **key management** are the three types of stored data necessary for developers to reliably build and deploy projects. + +``` +─ awesome-near-project ⟵ NEAR dApp + ├── .near-config ⟵ Stores project-level settings and connection configuration + │ ├── connections ⟵ Stores values needed to connect and deploy a contract, except keys + │ │ ├── default.js ⟵ Default configuration is for development + │ │ └── localnet.js ⟵ Custom connection added by user for localnet + │ └── settings.js ⟵ Stores near-shell settings, how shell behaves when run in this project + └── .near-credentials ⟵ Previously "neardev" this directory contains private key inforamtion + └── default + └── alice.json ⟵ NEAR account "alice" has private key information here +``` ### Definitions: @@ -32,26 +44,51 @@ These are key-values reflecting how `near-shell` behaves when called within a NE Besides answers to user prompts, project-level settings also store information about the last version of `near-shell` used in this project. For more information, please see [Upgradability](#upgradability). +As shown in the directory structure above, this file is located in the project directory at: + +`.near-config/settings.js` + In summary, "settings" are key-value pairs that are set by `near-shell` and relate to the behavior of how it operates in a given project directory. They do not relate to the development or deployment of smart contracts. #### Configuration Configuration are key-value pairs that relate to development and deployment of smart contracts and refer to *connection* information. +**Example**: As a user, I want to deploy a contract to my localnet instead of testnet during development. I will provide flags and/or modify configuration so that `near-shell` can determine where to connect, for what contract, and on behalf of which NEAR account. + +Contains: + * `networkId` : Similar to an environment to work on (ex: 'staging') * `nodeUrl` : URL to the NEAR node * `contractName` : NEAR account name for the smart contract * `walletUrl` : URL to NEAR Wallet * `helperUrl` : URL to the contract helper that provides tokens upon account creation -* `masterAccount` : When creating "child" accounts, `masterAccount` is the "parent" -* `configFile` : Specifies the location of a config file containing connection information. Default location is `src/config.js` - -**Example**: As a user, I want to deploy a contract to my localnet instead of testnet during development. I will provide flags and/or modify configuration so that `near-shell` can determine where to connect, for what contract, and on behalf of which NEAR account. +* `masterAccount` : When creating "child" accounts, `masterAccount` is the "parent" and source of initial Ⓝ balance +* `configFile` : Explicitly specifies the location of the config file containing connection information. Default location is `.near-config/connections/default.js` + +As shown in the directory structure earlier, this file is located in the project directory at: + +`.near-config/connections/default.js` + +**Note**: at the time of this writing, configuration is read from the project level at: + +`src/config.js` + +This is a file containing several environment configurations with a default, which the user can choose by setting the environment variable `NODE_ENV`. Configurations will no longer be set in that manner, but stored in files. + +For example, the command: + +`near contract view near-game topPlayers --env localnet` + +will call the function `topPlayers` on the contract `near-game` that is deployed to localnet. The connection information will be read from the file: `.near-config/connections/localnet.js`. + +Users may add, remove, or modify connection environments using [commands detailed later](#commands-config) in this spec. #### Key management +[key-management]: #key-management The storage of an account id and a corresponding private key. -* `keyPath` : An argument used by `near-shell` specifying the path to the keyfile. +* `keyPath` : An argument used by `near-shell` specifying the path to the key file. -The keyfile file contains the keys: +The key file file contains the keys: * `type` : Options include - `unencrypted` : An unencrypted file containing an account's private key - `native_osx` : Private key stored with OS X key management @@ -63,15 +100,12 @@ The keyfile file contains the keys: `near-shell` will look for keys in a specific order. This list is in the prioritized order and can be understood to mean, "if the key is not found here, then try the next location/store." 1. Environment variables: - - `NEAR_ACCOUNT_ID` - - `NEAR_ACCOUNT_TYPE` - - `NEAR_PRIVATE_KEY` - +* `NEAR_ACCOUNT_ID` +* `NEAR_ACCOUNT_TYPE` +* `NEAR_PRIVATE_KEY` 2. Project directory (`/Users/friend/projects/my-awesome-app/.near-credentials`) - **Note**: Formerly, the `neardev` folder contained the key files for a project. It is now `.near-credentials`. - 3. Home directory (`/Users/friend/.near-credentials`) If the key type among the `native_*` values, the operating system key management handles: @@ -84,7 +118,9 @@ This prioritized order allows project-level configuration to take priority over As an example, a user having credentials saved in their home directory will be able to use the `--accountId` flag to use keys from any project regardless of location. Said another way, the user does not need to run `near login` inside each project in order to access the keys. -**Note**: as the project grows, developers in the NEAR Collective may choose to add integrations with password management applications or hosted key solutions. +Formerly, the `neardev` folder contained the key files for a project. It is now `.near-credentials`. This folder is not nested within `.near-config` deliberately. Having this folder with "credentials" at the top-level of a project makes it apparent that it contains sensitive information and should not be revisioned or archived. + +**Note**: as the project grows, developers in the NEAR Collective may choose to add integrations with password management applications or hosted key solutions. Hence, expect the number of `type` options to increase. ## Translation @@ -210,19 +246,23 @@ This category is used to create, select, and configure accounts on NEAR networks * `near account select` : Select an account from the list of locally configured accounts as the active account. * `near account login` : Log in the current active or specified account. * `near account logout` : Log out the current active or specified account. Essentially revoking a full access key. +* `near account secure` : Finds the key file for an account name, converts it to use OS-level key management via a command line wizard. * `near account --help` : Display help on account subcommands. - ### `near config ` +[commands-config]: #commands-config + * `near config` : List the location of active configuration file if it's loaded, or default config settings -* `near config set` : Set a key in the active configuration, or if using default, create a config file in the home directory with defaults and the specified key and value -* `near config clear` : Remove the [project-level settings](#project-level-settings) file - -### `near contract ` -* `near contract build` -* `near contract deploy` -* `near contract call` -* `near contract view` +* `near config add` : Set a key in the active configuration, or if using default, create a config file in the home directory with defaults and the specified key and value +* `near config remove` : Removes a configuration file. (Example: removes `.near-config/connections/localnet.js`) +* `near config wizard` : Runs through a command-line wizard for an environment, allowing user to modify the values. + +### `near contract ` + +* `near contract deploy` : Deploys the contract in the project directory (Default: `out/main.wasm`) +* `near contract call` : Calls a function that may mutate state +* `near contract view` : Calls a function that reads state, does not mutate +* `near contract estimate` : Estimates gas consumption of a function call on a contract * `near contract --help` ## Command input can be inline and file-based @@ -238,10 +278,11 @@ or defined in a file: `near call my_dapp my_function -f ./params.json` This `-f` or `--fromFile` argument is added to two commands: -1. `contract call` -2. `contract view` -Other commands may be added in the future. +1. `near contract call` +2. `near contract view` + +Reading from a file may be added to other commands in the future. ### User Stories @@ -387,3 +428,12 @@ Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information. + +# From here to there + +This section highlights actionable items needed to get from the current state of the project to this spec. It is not exhaustive, but rather can be used to jog memories and start the conversation on atomic tasks. + +* Remove `near build` : With multiple languages in the future and multi-contract dApps, we cannot reliably run a single command to build. It needs to be communicated that the user will have to run a command or build script (not using `near-shell`) for this. +* Keys stored in `neardev` need to have an additional key added: `"type": "unencrypted"` +* The `neardev` folder needs to be renamed to `.near-credentials` +* Add `--env` flag so that config will load proper file. From 60a822a144808b8d84035144e231b445ba8babf5 Mon Sep 17 00:00:00 2001 From: Mike Purvis Date: Wed, 15 Apr 2020 17:01:27 -0700 Subject: [PATCH 06/13] more user stories, mention future ideas, remove validator references --- text/0000-near-shell.md | 206 +++++++++++++++++----------------------- 1 file changed, 88 insertions(+), 118 deletions(-) diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md index 332019193..5dd7cb74d 100644 --- a/text/0000-near-shell.md +++ b/text/0000-near-shell.md @@ -63,12 +63,13 @@ Contains: * `walletUrl` : URL to NEAR Wallet * `helperUrl` : URL to the contract helper that provides tokens upon account creation * `masterAccount` : When creating "child" accounts, `masterAccount` is the "parent" and source of initial Ⓝ balance -* `configFile` : Explicitly specifies the location of the config file containing connection information. Default location is `.near-config/connections/default.js` As shown in the directory structure earlier, this file is located in the project directory at: `.near-config/connections/default.js` +If a command is run and there exists no file in the path mentioned above, `near-shell` will prompt the user, suggesting it be created. + **Note**: at the time of this writing, configuration is read from the project level at: `src/config.js` @@ -86,7 +87,6 @@ Users may add, remove, or modify connection environments using [commands detaile #### Key management [key-management]: #key-management The storage of an account id and a corresponding private key. -* `keyPath` : An argument used by `near-shell` specifying the path to the key file. The key file file contains the keys: * `type` : Options include @@ -122,6 +122,10 @@ Formerly, the `neardev` folder contained the key files for a project. It is now **Note**: as the project grows, developers in the NEAR Collective may choose to add integrations with password management applications or hosted key solutions. Hence, expect the number of `type` options to increase. +## Operating system agnostic + +`near-shell` will use dependencies like [shelljs](https://github.com/shelljs/shelljs) in order to streamline input/output and paths such that the experience is identical across operating systems. It will not rely on hardcoded backslashes or forward slashes that may break functionality on a different operating system. It will not use unsupported commands like `mkdir -p` that does not exist on Windows. + ## Translation It's important to invite the international community into developing with NEAR. Translation of user-facing content is essential for: @@ -190,14 +194,14 @@ An example of a migration script might be: const upgrade = async (lastPatchVersion) => { if (lastPatchVersion < 2) { // implement essential logic that changed from x.x.0 to x.x.1 - // Example: the new minor version 0.24.0 changes the neardev directory to .near-credentials - // Logic here that checks for the absense of .near-credentials, the existence of neardev, and renames accordingly + // Example: the new minor version 0.24.0 renames the neardev directory to .near-credentials + // Logic here that checks for the absence of .near-credentials, the existence of neardev, and renames accordingly } if (lastPatchVersion < 6) { // implement essential logic that changed from x.x.2 to x.x.6 // Example: all keys in the .near-credentials need an additional key for "type" - // Logic looping through adding new key to existing files + // Logic to loop through adding the new key to existing key files } … @@ -225,12 +229,14 @@ Example: The number of prompts will grow beyond what can be captured in this spec. A number of possible prompts would be: +- Do you want to set a default account (See: `near account select`) +- Do you want to set a default account (See: `near config select`) - Is this project on an OS expected to have a browser and UI - When creating an account, always fund new accounts with the NEAR contract helper - Always run migrations in this project when applicable - Never run migrations in this project -## `near-shell` Commands +## Commands ### `near account ` @@ -243,19 +249,20 @@ This category is used to create, select, and configure accounts on NEAR networks * `near account delete` : Delete an account or sub-account. * `near account create-key` : Delete an account or sub-account. * `near account revoke-key` : Delete an account or sub-account. -* `near account select` : Select an account from the list of locally configured accounts as the active account. +* `near account select` : Selects a default account from the list of accounts with keys. Stored in project-level settings with key `defaultAccount`. * `near account login` : Log in the current active or specified account. -* `near account logout` : Log out the current active or specified account. Essentially revoking a full access key. -* `near account secure` : Finds the key file for an account name, converts it to use OS-level key management via a command line wizard. -* `near account --help` : Display help on account subcommands. +* `near account stake` : Stake Ⓝ to a given staking contract +* `near account secure` : Finds the key file for an account name, converts it to use OS-level key management via a command line wizard/instructions. +* `near account --help` ### `near config ` [commands-config]: #commands-config * `near config` : List the location of active configuration file if it's loaded, or default config settings * `near config add` : Set a key in the active configuration, or if using default, create a config file in the home directory with defaults and the specified key and value -* `near config remove` : Removes a configuration file. (Example: removes `.near-config/connections/localnet.js`) -* `near config wizard` : Runs through a command-line wizard for an environment, allowing user to modify the values. +* `near config remove` : Removes a configuration file. (Example: `near config remove localnet` removes `.near-config/connections/localnet.js`) +* `near config select` : Selects a default connection configuration file from the list of those available. Stored in the project-level settings with key `defaultConnection`. Default is `default`. +* `near config wizard` : Runs through a command-line wizard for an environment, allowing user to modify the values. ### `near contract ` @@ -265,175 +272,138 @@ This category is used to create, select, and configure accounts on NEAR networks * `near contract estimate` : Estimates gas consumption of a function call on a contract * `near contract --help` -## Command input can be inline and file-based +### `near chain ` -Some commands in `near-shell` may become long and difficult to type on the command line. End users with a standard terminal application may have lengthy arguments that are better saved to a file. +* `near chain block` : Lookup and show block info by hash or height +* `near chain chunk` : Lookup and show chunk info for a block +* `near chain transaction` : Lookup and show transaction status and info by hash -Flags may be done inline (default): +### `near validators ` -`near call my_dapp my_function '{"key": "value_19"}'` +Shows information on current validators and fisherman in the environment detailed by the connection configuration. -or defined in a file: - -`near call my_dapp my_function -f ./params.json` +* `near validators next` : Next validators and fisherman +* `near validators prev` : Previous validators and fisherman +* `near validators proposals` : Current proposals, also shows prospective threshold -This `-f` or `--fromFile` argument is added to two commands: +### `near tool ` +1. `near contract call` +2. `near contract view` -etc. TBD. +Reading from a file may be added to other commands in the future. -# Reference-level explanation -[reference-level-explanation]: #reference-level-explanation +## Analytics -TODO: - move some stuff down here +Usage analytics will be used anonymously and purely to measure key performance indicators. Mixpanel will be the chosen integration at this time. No private data will ever be sent, only metrics on the command usage and possible crash reporting. -This is the technical portion of the NEP. Explain the design in sufficient detail that: +There will be the ability to opt-out as well. -- Its interaction with other features is clear. -- It is reasonably clear how the feature would be implemented. -- Corner cases are dissected by example. +### User Stories -The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. +These enhancements to `near-shell` aim to increase user and developer adoption by centralizing and optimizing all interactions with NEAR functionality. `near-shell` today is a thin and lightweight set of features which enable the use of each NEAR product offering. Specifically, `near-shell` is required to use NEAR products today. For example, to stake, send tokens, create and delete accounts, view public keys, build and deploy smart contracts, call smart contract methods, and log in through NEAR Wallet. Each of these features are critical to all NEAR products. ---- +If `near-shell` is required, it makes sense to enhance `near-shell` to include other features critical to user and developer interactions. The commands proposed above enhance `near-shell` to include control for both local and distributed development, i.e., using a local network for offline development vs. testnet as a live network used by others, and to be able to select and configure networks. -The programming language for `near-shell` has not been determined yet but there have been opinions regarding proper, future-proof implementation. +Here are a few user stories to demonstrate the utility of these enhancements. -Long term goals to keep in mind: -- Upgradable, but also able to lock a specific version. -- Where possible, avoid single point of failures. This is in regards to using a package manager in particular. - - An official installer for multiple operating systems may be advised here, where trusted OS or GPG keys can verify the shell. -- Able to traverse directory structures from all operating systems - - This applies particularly to Windows where backslashes are used in the default command prompt. -- Limited dependencies to abate unforeseen issues. +* As a user, I want the option to have my operating system prompt for my user password before deploying a contract. +* As a user, I want to be able to have multiple projects that use the same account (and corresponding key(s)) without having to login to each project. +* As a user, I want to be able to instructions for commands in my own language. +* As a user, I want to be able to come back to a project months later, upgrade tooling, and face no issues building and deploying. +* As a user, I want to be able to set defaults and load a file in order to reduce the length of my CLI command when calling my contract. +* As a user, I want to be able to estimate the gas cost for a particular contract call before deploying. +* As a user, I want to use `near-shell` on any operating system and experience the full set of features. # Drawbacks [drawbacks]: #drawbacks Why should we *not* do this? -* The download size of `near-shell` will increase. The amount of increase will be determined by the code and libraries required to add the ability to deploy a validator node. * The complexity of `near-shell` increases whenever a new command is added. -* Today, the configuration of the NEAR networks is instigated and controlled using the `nearcore` repository. -* Not all users will want to run a validator node. -* These proposals increases the dependency chain of `near-shell` to include Rust nearcore libraries and configuration. - +* This proposal may increase the dependency chain of `near-shell` when it comes to gas estimation, user prompts, and possibly using environment variables instead of JavaScript/JSON. # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives -## Why is this design the best in the space of possible designs? +>Why is this design the best in the space of possible designs? -Today is difficult to communicate and introduce distributed Web concepts and features to users and developers. To address this, NEAR developers and products should reduce the overall complexity of distributed Web application development and create a centralized interface to all NEAR distributed application features and functionality. +It's difficult to communicate and introduce distributed web concepts to users and developers. To address this, NEAR developers and products should reduce the overall complexity of dApp development and create a centralized interface for dApp features and functionality. -Current best design practices for blockchain, wallets, and distributed Web applications attempt to mitigate blockchain interaction complexities by performing all required steps for interaction and feature use by *delaying* complex interactions and *encapsulating* multiple steps into as few interactions as possible. The enhancements proposed here embrace multiple facets of NEAR features by including the most complicated feature interactions (i.e., configuring multiple networks, account creation and configuration, starting and managing NEAR validator nodes, etc.) and allow complex interactions with NEAR's blockchain features to be delayed until absolutely required, i.e., validator node configuration and deployment is available immediately, on-demand, only as needed, with no extra steps required to run a local network or an active distributed blockchain network. The user is not required to pull a separate repository, hence *delaying* a relatively complex product interaction and practically eliminating most of the confusion introduced by fetching and inspecting another NEAR github repository. +Current best design practices for blockchain, wallets, and distributed web applications attempt to mitigate blockchain interaction complexities by performing all required steps for interaction and feature use by *delaying* complex interactions and *encapsulating* multiple steps into as few interactions as possible. The enhancements proposed here embrace multiple facets of NEAR features by including the most complicated feature interactions (i.e., configuring multiple networks, account creation and configuration, etc.) and allow complex interactions with NEAR's blockchain features to be delayed until absolutely required. The majority of the time, the user is not required to pull a separate repository, hence *delaying* a relatively complex product interaction and practically eliminating most of the confusion introduced by fetching and inspecting another NEAR repository. -These `near-shell` enhancements *encapsulate* multiple features and processes required to deploy and configure multiple networks. +These `near-shell` enhancements *encapsulate* multiple features and processes required to deploy and configure multiple networks for smart contract development. +>What other designs have been considered and what is the rationale for not choosing them? -- What other designs have been considered and what is the rationale for not choosing them? +The current `near-shell` design is the default. An approach might be to not use a shell at all. Instead, NEAR might create GUI applications, e.g., electron or web server-based configuration management where the user starts a local application which provides a port towards which the user can point their local browser to configure NEAR accounts, networks, and nodes. -The current `near-shell` design is the default. Other designs might be ones that do not include the ability to configure and deploy active validator nodes. Also, another approach might be to not use a shell at all. Instead, NEAR might create GUI applications, e.g., electron or Web server-based configuration management where the user starts a local application which provides a port towards which the user can point their local Web browser to configure NEAR accounts, networks, and nodes. +>What is the impact of not doing this? -- What is the impact of not doing this? - -Users will not have a single application which enables configuration and management of all NEAR features. Instead, multiple repositories with disparate commands and configuration will be required to use NEAR Protocol's distributed Web blockchain. +Users will not be able to customize their experience with defaults. Users will be forced to rely on unencrypted key files instead of using an operating system's trusted management. Users may have to type long, arduous commands when calling contracts. Users will not be able to estimate gas. # Unresolved questions [unresolved-questions]: #unresolved-questions -- What parts of the design do you expect to resolve through the NEP process before this gets merged? - -TODO: - remove all the stuff about ~/.near +>What parts of the design do you expect to resolve through the NEP process before this gets merged? -NEAR developers must weigh in on the impact of including validator node configuration. Also, the `~/.near` directory is currently created by `nearcore`. Decisions must be made about whether or not NEAR's configuration should be managed by `near-shell` instead of whenever a validator node is executed. +Adding the [upgradability mechanism](#upgradability) is key to `near-shell` being future-proof and the project would benefit from immediate implementation. -- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +>What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP? -The management of the `~/.near` directory should be resolved when these enhancements are stabilized. Also, the selection of the categories of commands and the commands themselves will need to be fleshed out to optimize both user and developer experiences when interacting with NEAR Protocol's distributed Web blockchain platform offering. +Out of scope is smart contract execution and debugging. However, future versions of `near-shell` might include specific commands that enable the debugging and optimization of smart contract deployments. Gas estimation is the first step that will likely bring execution much closer in feasibility. -- What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP? +#### Additional unresolved thoughts/questions -Out of scope is smart contract development and debugging. However, future versions of `near-shell` might include specific commands that enable the debugging and optimization of smart contract deployments. +Where possible, it's best to avoid single point of failures. This is in regards to using a package manager in particular. An official installer for multiple operating systems may be advised here, where trusted OS or GPG keys can verify the shell. + +Key storage for the metrics will be in plain-text, which is suboptimal but low risk. +It's unclear if any dependencies needed for this NEP might open the project up to remote execution security vulnerabilities. There is no reason to expect this, but a full security audit has not been completed at the time of this writing. # Future possibilities [future-possibilities]: #future-possibilities -### `near network ` - -This command category is used to select and configure NEAR networks for a `near-shell` user. This category manipulates the project's configuration to allow a user to specify NEAR network details via the CLI instead of manually editing config files. - -Sub-commands for `network` include: - -* `near network list` : Display the current list of networks for the current instance of `near-shell`. -* `near network status` : Show the user's current network configuration. -* `near network select` : Select a NEAR network as the default network for subsequent `near-shell` commands as well as the network configuration for a local server validator node managed by this instance of `near-shell`. -* `near network add` : Add a network to the users's `~/.near` config files. -* `near network remove` : Remove a network from the `~/.near` config files. -* `near network monitor` : Interactively monitor one or more NEAR networks in a curses-like interface updated in a specified interval, default of 1s. -* `near network help` : Display help on network subcommands. - -Think about what the natural extension and evolution of your proposal would -be and how it would affect the project as a whole in a holistic -way. Try to use this section as a tool to more fully consider all possible -interactions with the project in your proposal. -Also consider how the this all fits into the roadmap for the project -and of the relevant sub-team. - -This is also a good place to "dump ideas", if they are out of scope for the -NEP you are writing but otherwise related. - -If you have tried and cannot think of any future possibilities, -you may simply state that you cannot think of anything. +At the time of this writing, users wishing to run a NEAR validator node [will use nearup](https://github.com/near/nearup). In the future, it might be beneficial to roll this into `near-shell` or borrow features and usability from `near-shell` to the validator node user experience. -Note that having something written down in the future-possibilities section -is not a reason to accept the current or a future NEP. Such notes should be -in the section on motivation or rationale in this or subsequent NEPs. -The section merely provides additional information. +Currently `near-shell` is written in JavaScript using [yargs](https://yargs.js.org/). There are benefits of different platforms. Another JavaScript option is call [oclif](https://oclif.io/) which would provide the ability to have verified installs per operating system based on GPG keys or OS X certificates. There's also been a discussion about rewriting this in Rust. # From here to there This section highlights actionable items needed to get from the current state of the project to this spec. It is not exhaustive, but rather can be used to jog memories and start the conversation on atomic tasks. * Remove `near build` : With multiple languages in the future and multi-contract dApps, we cannot reliably run a single command to build. It needs to be communicated that the user will have to run a command or build script (not using `near-shell`) for this. -* Keys stored in `neardev` need to have an additional key added: `"type": "unencrypted"` -* The `neardev` folder needs to be renamed to `.near-credentials` +* Keys stored in `neardev` need to have an additional key added: `"type": "unencrypted"`. +* The `neardev` folder needs to be renamed to `.near-credentials`. * Add `--env` flag so that config will load proper file. From 6b6e9ce2ab5f9cb36b69accae3970d8ea997a29b Mon Sep 17 00:00:00 2001 From: Mike Purvis Date: Wed, 15 Apr 2020 17:15:33 -0700 Subject: [PATCH 07/13] add info about home directory settings, esp regarding metrics, settings commands --- text/0000-near-shell.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md index 5dd7cb74d..5f09646c1 100644 --- a/text/0000-near-shell.md +++ b/text/0000-near-shell.md @@ -49,6 +49,8 @@ As shown in the directory structure above, this file is located in the project d `.near-config/settings.js` In summary, "settings" are key-value pairs that are set by `near-shell` and relate to the behavior of how it operates in a given project directory. They do not relate to the development or deployment of smart contracts. + +**Note**: project-level settings can exist in the user's home directory as well. In this case, projects without their own `.near-config/settings.js` will use the home directory settings. `near-shell` will be quite verbose when running a command, letting the user know which settings and configuration files are being used when the user runs commands. #### Configuration Configuration are key-value pairs that relate to development and deployment of smart contracts and refer to *connection* information. @@ -264,6 +266,12 @@ This category is used to create, select, and configure accounts on NEAR networks * `near config select` : Selects a default connection configuration file from the list of those available. Stored in the project-level settings with key `defaultConnection`. Default is `default`. * `near config wizard` : Runs through a command-line wizard for an environment, allowing user to modify the values. +### `near settings ` + +* `near settings` : Displays settings in project and home directory, if they exist +* `near settings set-default` : Copies the project's settings into the home directory's settings +* `near settings clear` : Prompts user, then removes select settings file (in project or home) + ### `near contract ` * `near contract deploy` : Deploys the contract in the project directory (Default: `out/main.wasm`) @@ -286,7 +294,7 @@ Shows information on current validators and fisherman in the environment detaile * `near validators prev` : Previous validators and fisherman * `near validators proposals` : Current proposals, also shows prospective threshold -### `near tool ` Utilities added to `near-shell` @@ -328,7 +336,7 @@ Reading from a file may be added to other commands in the future. Usage analytics will be used anonymously and purely to measure key performance indicators. Mixpanel will be the chosen integration at this time. No private data will ever be sent, only metrics on the command usage and possible crash reporting. -There will be the ability to opt-out as well. +There will be the ability to opt-out as well. While the majority of settings are project-level, this is an example of a setting that will be saved to the user's home directory settings. (Example: `~/.near-config/settings.js`) ### User Stories From 6a8adea3b76f13b10f30e6aa55507f942a442c89 Mon Sep 17 00:00:00 2001 From: Mike Purvis Date: Wed, 15 Apr 2020 18:16:08 -0700 Subject: [PATCH 08/13] clarify some settings are home-only & project-only --- text/0000-near-shell.md | 58 ++++++++++++++++++++++++++++++----------- 1 file changed, 43 insertions(+), 15 deletions(-) diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md index 5f09646c1..b99a3576b 100644 --- a/text/0000-near-shell.md +++ b/text/0000-near-shell.md @@ -25,7 +25,7 @@ Project-level **settings**, connection **configuration**, and **key management** ``` ─ awesome-near-project ⟵ NEAR dApp ├── .near-config ⟵ Stores project-level settings and connection configuration - │ ├── connections ⟵ Stores values needed to connect and deploy a contract, except keys + │ ├── connections ⟵ Contains files used to connect and deploy a contract (except keys) │ │ ├── default.js ⟵ Default configuration is for development │ │ └── localnet.js ⟵ Custom connection added by user for localnet │ └── settings.js ⟵ Stores near-shell settings, how shell behaves when run in this project @@ -38,9 +38,12 @@ Project-level **settings**, connection **configuration**, and **key management** #### Settings [project-level-settings]: #project-level-settings + +--- + These are key-values reflecting how `near-shell` behaves when called within a NEAR project. -**Example**: a developer using OS X uses `near login` with access to a browser, whereas a validator runs `near login` on a CentOS box with no UI or browser. A user prompt may ask "Is this login from a computer with a browser?" The answer is then stored in key-value format in the project-level directory so it can be referenced later, skipping the prompt in the future. +Example: a developer using OS X uses `near login` with access to a browser, whereas a validator runs `near login` on a CentOS box with no UI or browser. A user prompt may ask "Is this login from a computer with a browser?" The answer is then stored in key-value format in the project-level directory so it can be referenced later, skipping the prompt in the future. Besides answers to user prompts, project-level settings also store information about the last version of `near-shell` used in this project. For more information, please see [Upgradability](#upgradability). @@ -48,14 +51,33 @@ As shown in the directory structure above, this file is located in the project d `.near-config/settings.js` -In summary, "settings" are key-value pairs that are set by `near-shell` and relate to the behavior of how it operates in a given project directory. They do not relate to the development or deployment of smart contracts. +In summary, "settings" are key-value pairs that are set by `near-shell` and relate to the behavior of how it operates in a given project directory. They do not relate to the development or deployment of smart contracts, except for providing default values like `accountId`. **Note**: project-level settings can exist in the user's home directory as well. In this case, projects without their own `.near-config/settings.js` will use the home directory settings. `near-shell` will be quite verbose when running a command, letting the user know which settings and configuration files are being used when the user runs commands. + +To make this more clear, there are two nested arrays that differentiate settings that should only be applied from a project-level file from those in the home directory. This is how a `settings.js` file may look: + +```javascript +{ + … + "alwaysRunMigrations": true, + "project-only": { + "lastShellVersion": "0.19.1" + } +} +``` + +As defined later, there is a command `near settings set-default` which copies the current project's settings into the home directory. Keys inside the `project-only` object will not be moved. + +If a new project is created and has no project-level settings yet and the home directory has a settings file, this will be used as the template. Otherwise, a project-level settings file will be created and populated based on the user prompts given during regular usage of `near-shell`. #### Configuration + +--- + Configuration are key-value pairs that relate to development and deployment of smart contracts and refer to *connection* information. -**Example**: As a user, I want to deploy a contract to my localnet instead of testnet during development. I will provide flags and/or modify configuration so that `near-shell` can determine where to connect, for what contract, and on behalf of which NEAR account. +Example: As a user, I want to deploy a contract to my localnet instead of testnet during development. I will provide flags and/or modify configuration so that `near-shell` can determine where to connect, for what contract, and on behalf of which NEAR account. Contains: @@ -84,10 +106,13 @@ For example, the command: will call the function `topPlayers` on the contract `near-game` that is deployed to localnet. The connection information will be read from the file: `.near-config/connections/localnet.js`. -Users may add, remove, or modify connection environments using [commands detailed later](#commands-config) in this spec. +Users may add, remove, or modify connection environments using commands detailed later in this spec. #### Key management [key-management]: #key-management + +--- + The storage of an account id and a corresponding private key. The key file file contains the keys: @@ -126,7 +151,7 @@ Formerly, the `neardev` folder contained the key files for a project. It is now ## Operating system agnostic -`near-shell` will use dependencies like [shelljs](https://github.com/shelljs/shelljs) in order to streamline input/output and paths such that the experience is identical across operating systems. It will not rely on hardcoded backslashes or forward slashes that may break functionality on a different operating system. It will not use unsupported commands like `mkdir -p` that does not exist on Windows. +`near-shell` will use dependencies like [shelljs](https://github.com/shelljs/shelljs) in order to streamline input/output and paths such that the experience is identical across operating systems. It will not rely on hardcoded backslashes or forward slashes that may break functionality on a different operating system. It will not use unsupported commands like `mkdir -p` that does not exist on Windows, etc. ## Translation @@ -136,9 +161,9 @@ It's important to invite the international community into developing with NEAR. Language preference can be set in two ways: - Using environment variables (i.e., parsed from `process.env.LANG`, and the default assignment) -- Set explicitly in [project-level settings](#project-level-settings) using [ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) +- Set explicitly in project-level settings using [ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) -Example content from `/.near-config/settings.env`: +Example content from `/.near-config/settings.js`: ```json … "language": "en" @@ -152,11 +177,11 @@ Commands and subcommands (ex: `call` or `deploy`) are not translated and will be As `near-shell` matures, updates may/will cause a user's project to become outdated. For instance, this NEP proposes to rename the project directory `neardev` to `.near-credentials`. Developers with existing `neardev` directories will either need to manually rename the folder, or `near-shell` will have to look in two places for the key files. -This proposal declares that `near-shell` will have a mechanism to make such upgrades possible. This is not possible, however, without keeping track of which version of `near-shell` was most recently used on a project. +This proposal declares that `near-shell` will have a mechanism to make such upgrades possible. This endeavor cannot happen, however, without keeping track of which version of `near-shell` was most recently used in a project. -**Example scenario**: a user begins developing a dApp on NEAR using `near-shell` version 0.19.0. The user takes a sabbatical for a few months and returns with a new computer, cloning the old project. The new computer installed `near-shell` version 0.23.1. The first time this user runs a command in this old project, migrations are run ensuring the project stays current. +**Example scenario**: a user begins developing a dApp on NEAR using `near-shell` version `0.19.0`. The user takes a sabbatical for a few months and returns with a new computer, cloning the old project. The new computer installed `near-shell` version `0.23.1`. The first time this user runs a command in this old project, migrations are run ensuring the project stays current. -Of the three types of storage mentioned before (project-level settings, connection configuration, and key management) the saved version information belongs to the project-level settings. It's possible that a user will have multiple dApps that are used less frequently than others. Each project must have their own record of which version of `near-shell` was most recently used. +Of the three types of storage mentioned before (project-level settings, connection configuration, and key management) the saved version information belongs to the project-level settings. It's possible that a user will have multiple dApps that are used less frequently than others and become out of date. Therefore, each project must have their own record of which version of `near-shell` was most recently used. This setting will never be placed in the home directory. The location of these settings is: @@ -165,6 +190,8 @@ The location of these settings is: For example: `/Users/friend/near-projects/guest-book/.near-config/settings.js` +Currently, this is the only setting that belongs to the object `project-only` in the settings file. The setting `lastShellVersion` will never be loaded from the home directory. + In this proposal, the file is shown as JavaScript. This is not a hard requirement, as as key-value pairs could also be in the form of environment variables: ```bash @@ -258,7 +285,6 @@ This category is used to create, select, and configure accounts on NEAR networks * `near account --help` ### `near config ` -[commands-config]: #commands-config * `near config` : List the location of active configuration file if it's loaded, or default config settings * `near config add` : Set a key in the active configuration, or if using default, create a config file in the home directory with defaults and the specified key and value @@ -332,11 +358,11 @@ This `-f` or `--fromFile` argument is added to two commands: Reading from a file may be added to other commands in the future. -## Analytics +## Analytics / Metrics -Usage analytics will be used anonymously and purely to measure key performance indicators. Mixpanel will be the chosen integration at this time. No private data will ever be sent, only metrics on the command usage and possible crash reporting. +Usage analytics will be used anonymously and purely to measure key performance indicators. [Mixpanel](https://mixpanel.com/) will be the chosen integration at this time. No private data will ever be sent, only metrics on the command usage and possible crash reporting. -There will be the ability to opt-out as well. While the majority of settings are project-level, this is an example of a setting that will be saved to the user's home directory settings. (Example: `~/.near-config/settings.js`) +There will be the ability to opt-out as well. While the majority of settings are project-level, this is the only setting that will be saved directly to the user's home directory settings instead of the project-level settings file. (That is, stored in: `~/.near-config/settings.js`) ### User Stories @@ -392,6 +418,8 @@ Adding the [upgradability mechanism](#upgradability) is key to `near-shell` bein Out of scope is smart contract execution and debugging. However, future versions of `near-shell` might include specific commands that enable the debugging and optimization of smart contract deployments. Gas estimation is the first step that will likely bring execution much closer in feasibility. +In this proposal, `near account select` and `near config select` add to the project-level settings. These could arguably exist in the connection configuration. + #### Additional unresolved thoughts/questions Where possible, it's best to avoid single point of failures. This is in regards to using a package manager in particular. An official installer for multiple operating systems may be advised here, where trusted OS or GPG keys can verify the shell. From 8e13193352d766bb6ffd8b2180c7a72b62ea77f1 Mon Sep 17 00:00:00 2001 From: Mike Purvis Date: Wed, 15 Apr 2020 18:28:26 -0700 Subject: [PATCH 09/13] differentiate shell-experience & project-level settings --- text/0000-near-shell.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md index b99a3576b..0edf259bd 100644 --- a/text/0000-near-shell.md +++ b/text/0000-near-shell.md @@ -20,11 +20,11 @@ These enhancements are intended to simplify and enhance developer and user produ ## Settings, config, and key management -Project-level **settings**, connection **configuration**, and **key management** are the three types of stored data necessary for developers to reliably build and deploy projects. +Shell-experience **settings**, connection **configuration**, and **key management** are the three types of stored data necessary for developers to reliably build and deploy projects. ``` ─ awesome-near-project ⟵ NEAR dApp - ├── .near-config ⟵ Stores project-level settings and connection configuration + ├── .near-config ⟵ Stores shell-experience settings and connection configuration │ ├── connections ⟵ Contains files used to connect and deploy a contract (except keys) │ │ ├── default.js ⟵ Default configuration is for development │ │ └── localnet.js ⟵ Custom connection added by user for localnet @@ -37,7 +37,7 @@ Project-level **settings**, connection **configuration**, and **key management** ### Definitions: #### Settings -[project-level-settings]: #project-level-settings +[shell-experience-settings]: #shell-experience-settings --- @@ -45,7 +45,7 @@ These are key-values reflecting how `near-shell` behaves when called within a NE Example: a developer using OS X uses `near login` with access to a browser, whereas a validator runs `near login` on a CentOS box with no UI or browser. A user prompt may ask "Is this login from a computer with a browser?" The answer is then stored in key-value format in the project-level directory so it can be referenced later, skipping the prompt in the future. -Besides answers to user prompts, project-level settings also store information about the last version of `near-shell` used in this project. For more information, please see [Upgradability](#upgradability). +Besides answers to user prompts, shell-experience settings also store information about the last version of `near-shell` used in this project. For more information, please see [Upgradability](#upgradability). As shown in the directory structure above, this file is located in the project directory at: @@ -53,7 +53,7 @@ As shown in the directory structure above, this file is located in the project d In summary, "settings" are key-value pairs that are set by `near-shell` and relate to the behavior of how it operates in a given project directory. They do not relate to the development or deployment of smart contracts, except for providing default values like `accountId`. -**Note**: project-level settings can exist in the user's home directory as well. In this case, projects without their own `.near-config/settings.js` will use the home directory settings. `near-shell` will be quite verbose when running a command, letting the user know which settings and configuration files are being used when the user runs commands. +**Note**: shell-experience settings can exist in the user's home directory as well. In this case, projects without their own `.near-config/settings.js` will use the home directory settings. `near-shell` will be quite verbose when running a command, letting the user know which settings and configuration files are being used when the user runs commands. To make this more clear, there are two nested arrays that differentiate settings that should only be applied from a project-level file from those in the home directory. This is how a `settings.js` file may look: @@ -69,7 +69,7 @@ To make this more clear, there are two nested arrays that differentiate settings As defined later, there is a command `near settings set-default` which copies the current project's settings into the home directory. Keys inside the `project-only` object will not be moved. -If a new project is created and has no project-level settings yet and the home directory has a settings file, this will be used as the template. Otherwise, a project-level settings file will be created and populated based on the user prompts given during regular usage of `near-shell`. +If a new project is created and has no project-level settings file yet, and the home directory has a settings file, the home directory's file will be used as the template. Otherwise, a project-level settings file will be created and populated based on the user prompts given during regular usage of `near-shell`. #### Configuration @@ -161,7 +161,7 @@ It's important to invite the international community into developing with NEAR. Language preference can be set in two ways: - Using environment variables (i.e., parsed from `process.env.LANG`, and the default assignment) -- Set explicitly in project-level settings using [ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) +- Set explicitly in shell-experience settings using [ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) Example content from `/.near-config/settings.js`: ```json @@ -181,7 +181,7 @@ This proposal declares that `near-shell` will have a mechanism to make such upgr **Example scenario**: a user begins developing a dApp on NEAR using `near-shell` version `0.19.0`. The user takes a sabbatical for a few months and returns with a new computer, cloning the old project. The new computer installed `near-shell` version `0.23.1`. The first time this user runs a command in this old project, migrations are run ensuring the project stays current. -Of the three types of storage mentioned before (project-level settings, connection configuration, and key management) the saved version information belongs to the project-level settings. It's possible that a user will have multiple dApps that are used less frequently than others and become out of date. Therefore, each project must have their own record of which version of `near-shell` was most recently used. This setting will never be placed in the home directory. +Of the three types of storage mentioned before (shell-experience settings, connection configuration, and key management) the saved version information belongs to the project-level settings. It's possible that a user will have multiple dApps that are used less frequently than others and become out of date. Therefore, each project must have their own record of which version of `near-shell` was most recently used. This setting will never be placed in the home directory. The location of these settings is: @@ -278,7 +278,7 @@ This category is used to create, select, and configure accounts on NEAR networks * `near account delete` : Delete an account or sub-account. * `near account create-key` : Delete an account or sub-account. * `near account revoke-key` : Delete an account or sub-account. -* `near account select` : Selects a default account from the list of accounts with keys. Stored in project-level settings with key `defaultAccount`. +* `near account select` : Selects a default account from the list of accounts with keys. Stored in shell-experience settings with key `defaultAccount`. * `near account login` : Log in the current active or specified account. * `near account stake` : Stake Ⓝ to a given staking contract * `near account secure` : Finds the key file for an account name, converts it to use OS-level key management via a command line wizard/instructions. @@ -289,7 +289,7 @@ This category is used to create, select, and configure accounts on NEAR networks * `near config` : List the location of active configuration file if it's loaded, or default config settings * `near config add` : Set a key in the active configuration, or if using default, create a config file in the home directory with defaults and the specified key and value * `near config remove` : Removes a configuration file. (Example: `near config remove localnet` removes `.near-config/connections/localnet.js`) -* `near config select` : Selects a default connection configuration file from the list of those available. Stored in the project-level settings with key `defaultConnection`. Default is `default`. +* `near config select` : Selects a default connection configuration file from the list of those available. Stored in the shell-experience settings with key `defaultConnection`. Default is `default`. * `near config wizard` : Runs through a command-line wizard for an environment, allowing user to modify the values. ### `near settings ` @@ -362,7 +362,7 @@ Reading from a file may be added to other commands in the future. Usage analytics will be used anonymously and purely to measure key performance indicators. [Mixpanel](https://mixpanel.com/) will be the chosen integration at this time. No private data will ever be sent, only metrics on the command usage and possible crash reporting. -There will be the ability to opt-out as well. While the majority of settings are project-level, this is the only setting that will be saved directly to the user's home directory settings instead of the project-level settings file. (That is, stored in: `~/.near-config/settings.js`) +There will be the ability to opt-out as well. While the majority of shell-experience settings are project-level, this is the only setting that will be saved directly to the user's home directory settings instead of the project-level settings file. (That is, stored in: `~/.near-config/settings.js`) ### User Stories @@ -418,7 +418,7 @@ Adding the [upgradability mechanism](#upgradability) is key to `near-shell` bein Out of scope is smart contract execution and debugging. However, future versions of `near-shell` might include specific commands that enable the debugging and optimization of smart contract deployments. Gas estimation is the first step that will likely bring execution much closer in feasibility. -In this proposal, `near account select` and `near config select` add to the project-level settings. These could arguably exist in the connection configuration. +In this proposal, `near account select` and `near config select` add to the shell-experience settings. These could arguably exist in the connection configuration. #### Additional unresolved thoughts/questions @@ -443,3 +443,4 @@ This section highlights actionable items needed to get from the current state of * Keys stored in `neardev` need to have an additional key added: `"type": "unencrypted"`. * The `neardev` folder needs to be renamed to `.near-credentials`. * Add `--env` flag so that config will load proper file. +* Begin implementing yargs' [locale ability](https://yargs.js.org/docs/#api-locale) for translation. From 974887126800be47a58e968fac5b997db7d50f7d Mon Sep 17 00:00:00 2001 From: Mike Purvis Date: Wed, 15 Apr 2020 18:59:58 -0700 Subject: [PATCH 10/13] minor corrections, introductory paragraph, remove incorrect metrics info --- text/0000-near-shell.md | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md index 0edf259bd..e77cb833c 100644 --- a/text/0000-near-shell.md +++ b/text/0000-near-shell.md @@ -18,6 +18,8 @@ These enhancements are intended to simplify and enhance developer and user produ # Guide-level explanation [guide-level-explanation]: #guide-level-explanation +This NEP will jump straight into the concepts necessary to understand the components needed in `near-shell`, and generally an extensible CLI tool. Unlike other NEPs, the guide-level and reference-level explanations are combined. There is nothing overly technical in the following sections, but more conceptual in nature. With that, the discussion begins by separating the three types of stored information needed for a successful, long-term `near-shell`. + ## Settings, config, and key management Shell-experience **settings**, connection **configuration**, and **key management** are the three types of stored data necessary for developers to reliably build and deploy projects. @@ -36,7 +38,7 @@ Shell-experience **settings**, connection **configuration**, and **key managemen ### Definitions: -#### Settings +#### 1. Settings [shell-experience-settings]: #shell-experience-settings --- @@ -71,7 +73,7 @@ As defined later, there is a command `near settings set-default` which copies th If a new project is created and has no project-level settings file yet, and the home directory has a settings file, the home directory's file will be used as the template. Otherwise, a project-level settings file will be created and populated based on the user prompts given during regular usage of `near-shell`. -#### Configuration +#### 2. Configuration --- @@ -108,7 +110,7 @@ will call the function `topPlayers` on the contract `near-game` that is deployed Users may add, remove, or modify connection environments using commands detailed later in this spec. -#### Key management +#### 3. Key management [key-management]: #key-management --- @@ -221,31 +223,31 @@ An example of a migration script might be: ```javascript const upgrade = async (lastPatchVersion) => { - if (lastPatchVersion < 2) { - // implement essential logic that changed from x.x.0 to x.x.1 - // Example: the new minor version 0.24.0 renames the neardev directory to .near-credentials - // Logic here that checks for the absence of .near-credentials, the existence of neardev, and renames accordingly - } + if (lastPatchVersion < 2) { + // implement essential logic that changed from x.x.0 to x.x.1 + // Example: the new minor version 0.24.0 renames the neardev directory to .near-credentials + // Logic here that checks for the absence of .near-credentials, the existence of neardev, and renames accordingly + } - if (lastPatchVersion < 6) { - // implement essential logic that changed from x.x.2 to x.x.6 - // Example: all keys in the .near-credentials need an additional key for "type" - // Logic to loop through adding the new key to existing key files - } + if (lastPatchVersion < 6) { + // implement essential logic that changed from x.x.2 to x.x.6 + // Example: all keys in the .near-credentials need an additional key for "type" + // Logic to loop through files adding the new key + } - … + … }; exports.upgrade = upgrade; ``` -Shown in the directory structure above is the file `shell-upgrade.js`. This file will, after comparing the current version to the last used version in `./.near-config/settings.js`, determine how many migration scripts are needed to run. It will loop through the necessary files calling the `upgrade()` function on them in the proper version order. When complete, it will update the `lastShellVersion` key in the project-level settings file. At this time the project is considered current. +Shown in the directory structure above is the file `shell-upgrade.js`. This file will, after comparing the current version to the last used version in the project's `.near-config/settings.js` file, determine how many migration scripts are needed to run. It will loop through the necessary files calling the `upgrade()` function on them in the proper version order. When complete, it will update the `lastShellVersion` key in the project-level settings file. At this time the project is considered current. **Note**: migrations do not have to fix backwards-incompatible changes. Migrations can also improve experience by, for example, removing orphaned files, make safety checks and show warnings, etc. ## Prompts -Various user prompts will enhance the experience of `near-shell` by providing options. These prompts have the option to save answers to that identical future prompts may be skipped if desired. As mentioned, the answers will be saved as **settings** on the project level. +Various user prompts will enhance the experience of `near-shell` by providing options. These prompts can save answers so that they may be skipped in the future. As mentioned, the answers will be saved as **settings** on the project level. Example: `/Users/friend/near-projects/.near-config/settings.js` has: @@ -322,10 +324,10 @@ Shows information on current validators and fisherman in the environment detaile ### `near tool ` -Utilities added to `near-shell` +Utility tools added to `near-shell` * `near tool repl` : Read evaluate print loop tool -* `near tool metrics` : Allows user to opt-out or opt-in from metrics. (Default: opt-in) +* `near tool metrics` : Allows user to opt-out or opt-in from metrics ### Flags From 5bc61e292faf711d275fc7955a950015f8f8811c Mon Sep 17 00:00:00 2001 From: Mike Purvis Date: Wed, 22 Apr 2020 13:56:35 -0700 Subject: [PATCH 11/13] add "network" key to key file --- text/0000-near-shell.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md index e77cb833c..21f499627 100644 --- a/text/0000-near-shell.md +++ b/text/0000-near-shell.md @@ -124,6 +124,7 @@ The key file file contains the keys: - `native_linux` : Private key stored with Linux-based system's key management - `native_windows` : Private key stored with Windows key management * `account_id` : The NEAR account name +* `network` : The NEAR network (example, "betanet", "testnet", etc.) * `private_key` : The plain-text private key, used when `type = "unencrypted"` `near-shell` will look for keys in a specific order. This list is in the prioritized order and can be understood to mean, "if the key is not found here, then try the next location/store." From 31a7f2d2f8f495c3b594c60450bae244796639e5 Mon Sep 17 00:00:00 2001 From: Mike Purvis Date: Mon, 27 Apr 2020 17:51:00 -0700 Subject: [PATCH 12/13] by default key files are stored in the home directory --- text/0000-near-shell.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md index 21f499627..d3570961a 100644 --- a/text/0000-near-shell.md +++ b/text/0000-near-shell.md @@ -115,7 +115,7 @@ Users may add, remove, or modify connection environments using commands detailed --- -The storage of an account id and a corresponding private key. +The storage of an account id and a corresponding private key. When a user runs `near account login` the default destination is the home directory. This can be overridden as explained in a moment. The key file file contains the keys: * `type` : Options include @@ -282,7 +282,7 @@ This category is used to create, select, and configure accounts on NEAR networks * `near account create-key` : Delete an account or sub-account. * `near account revoke-key` : Delete an account or sub-account. * `near account select` : Selects a default account from the list of accounts with keys. Stored in shell-experience settings with key `defaultAccount`. -* `near account login` : Log in the current active or specified account. +* `near account login` : Log in the current active or specified account. The default location of unencrypted key files is the home directory. * `near account stake` : Stake Ⓝ to a given staking contract * `near account secure` : Finds the key file for an account name, converts it to use OS-level key management via a command line wizard/instructions. * `near account --help` From 87463e7a551885ccdcacf9c141881a9cef1968d0 Mon Sep 17 00:00:00 2001 From: Mike Purvis Date: Thu, 7 May 2020 18:48:09 -0700 Subject: [PATCH 13/13] distinguish validators and proposals command --- text/0000-near-shell.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/text/0000-near-shell.md b/text/0000-near-shell.md index d3570961a..7f029fd23 100644 --- a/text/0000-near-shell.md +++ b/text/0000-near-shell.md @@ -321,7 +321,10 @@ Shows information on current validators and fisherman in the environment detaile * `near validators next` : Next validators and fisherman * `near validators prev` : Previous validators and fisherman -* `near validators proposals` : Current proposals, also shows prospective threshold + +### `near proposals ` + +* `near proposals` : Current proposals by epoch ### `near tool `