storage-node: update READMEs

storage-node/README.md

@@ -4,11 +4,11 @@ This repository contains several Node packages, located under the `packages/`
 subdirectory. See each individual package for details:
 
 * [colossus](./packages/colossus/README.md) - the main colossus app.
-* [storage](./packages/storage/README.md) - abstraction over the storage backend.
-* [runtime-api](./packages/runtime-api/README.md) - convenience wrappers for the runtime API.
-* [crypto](./packages/crypto/README.md) - cryptographic utility functions.
-* [util](./packages/util/README.md) - general utility functions.
+* [storage-node-backend](./packages/storage/README.md) - abstraction over the storage backend.
+* [storage-runtime-api](./packages/runtime-api/README.md) - convenience wrappers for the runtime API.
+* [storage-utils](./packages/util/README.md) - general utility functions.
 * [discovery](./packages/discovery/README.md) - service discovery using IPNS.
+* [storage-cli](./packages/cli/README.md) - cli for uploading and downloading content from the network
 
 Installation
 ------------
@@ -40,17 +40,48 @@ $ yarn install
 The command will install dependencies, and make a `colossus` executable available:
 
 ```bash
-$ yarn run colossus --help
+$ yarn colossus --help
 ```
 
 *Testing*
 
-Running tests from the repository root will run tests from all packages:
+Run an ipfs node and a joystream-node development chain (in separate terminals)
 
+```sh
+ipfs daemon
 ```
-$ yarn run test
+
+```sh
+joystream-node --dev
+```
+
+```sh
+$ yarn workspace storage-node test
+```
+
+Running a development environment, after starting the ipfs node and development chain
+
+```sh
+yarn storage-cli dev-init
+```
+
+This will configure the running chain with alice as the storage lead and with a know role key for
+the storage provider.
+
+Run colossus in development mode:
+
+```sh
+yarn colossus --dev
+```
+
+Start pioneer ui:
+``sh
+yarn workspace pioneer start
 ```
 
+Browse pioneer on http://localhost:3000/
+You should find Alice account is the storage working group lead and is a storage provider
+Create a media channel. And upload a file.
 
 ## Detailed Setup and Configuration Guide
 For details on how to setup a storage node on the Joystream network, follow this [step by step guide](https://github.com/Joystream/helpdesk/tree/master/roles/storage-providers).

storage-node/license_header.txt

@@ -1,18 +0,0 @@
-/*
- * This file is part of the storage node for the Joystream project.
- * Copyright (C) 2019 Joystream Contributors
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation, either version 3 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program.  If not, see <https://www.gnu.org/licenses/>.
- */
-

storage-node/packages/cli/README.md

@@ -1,5 +1,40 @@
 # A CLI for the Joystream Runtime & Colossus
 
-- CLI access for some functionality from `@joystream/storage-runtime-api`
+- CLI access for some functionality from other packages in the storage-node workspace
 - Colossus/storage node functionality:
   - File uploads
+  - File downloads
+- Development
+  - Setup development environment
+
+Running the storage cli tool:
+
+```sh
+$ yarn storage-cli --help
+```
+
+```sh
+
+  Joystream tool for uploading and downloading files to the network
+
+  Usage:
+    $ storage-cli command [arguments..] [key_file] [passphrase]
+
+  Some commands require a key file as the last option holding the identity for
+  interacting with the runtime API.
+
+  Commands:
+    upload            Upload a file to a Colossus storage node. Requires a
+                      storage node URL, and a local file name to upload. As
+                      an optional third parameter, you can provide a Data
+                      Object Type ID - this defaults to "1" if not provided.
+    download          Retrieve a file. Requires a storage node URL and a content
+                      ID, as well as an output filename.
+    head              Send a HEAD request for a file, and print headers.
+                      Requires a storage node URL and a content ID.
+
+  Dev Commands:       Commands to run on a development chain.
+    dev-init          Setup chain with Alice as lead and storage provider.
+    dev-check         Check the chain is setup with Alice as lead and storage provider.
+
+```

storage-node/packages/colossus/README.md

@@ -3,59 +3,53 @@
 Development
 -----------
 
-Run a development server:
+Run a development server (an ipfs node and development chain should be running on the local machine)
 
 ```bash
-$ yarn run dev --config myconfig.json
+$ yarn colossus --dev
 ```
 
-Command-Line
-------------
+This will expect the chain to be configured with certain development accounts.
+The setup can be done by running the dev-init command for the storage-cli:
 
-Running a storage server is (almost) as easy as running the bundled `colossus`
-executable:
-
-```bash
-$ colossus --storage=/path/to/storage/directory
+```sh
+yarn storage-cli dev-init
 ```
 
-Run with `--help` to see a list of available CLI options.
-
-You need to stake as a storage provider to run a storage node.
-
-Configuration
--------------
-
-Most common configuration options are available as command-line options
-for the CLI.
 
-However, some advanced configuration options are only possible to set
-via the configuration file.
-
-* `filter` is a hash of upload filtering options.
-  * `max_size` sets the maximum permissible file upload size. If unset,
-    this defaults to 100 MiB.
-  * `mime` is a hash of...
-    * `accept` is an Array of mime types that are acceptable for uploads,
-      such as `text/plain`, etc. Mime types can also be specified for
-      wildcard matching, such as `video/*`.
-    * `reject` is an Array of mime types that are unacceptable for uploads.
-
-Upload Filtering
-----------------
+Command-Line
+------------
 
-The upload filtering logic first tests whether any of the `accept` mime types
-are matched. If none are matched, the upload is rejected. If any is matched,
-then the upload is still rejected if any of the `reject` mime types are
-matched.
+```sh
+$ yarn colossus --help
+```
 
-This allows inclusive and exclusive filtering.
+```
+  Colossus - Joystream Storage Node
+
+  Usage:
+    $ colossus [command] [arguments]
+
+  Commands:
+    server        Runs a production server instance. (discovery and storage services)
+                  This is the default command if not specified.
+    discovery     Run the discovery service only.
+
+  Arguments (required for server. Ignored if running server with --dev option):
+    --provider-id ID, -i ID     StorageProviderId assigned to you in working group.
+    --key-file FILE             JSON key export file to use as the storage provider (role account).
+    --public-url=URL, -u URL    API Public URL to announce.
+
+  Arguments (optional):
+    --dev                   Runs server with developer settings.
+    --passphrase            Optional passphrase to use to decrypt the key-file.
+    --port=PORT, -p PORT    Port number to listen on, defaults to 3000.
+    --ws-provider WS_URL    Joystream-node websocket provider, defaults to ws://localhost:9944
+```
 
-* `{ accept: ['text/plain', 'text/html'] }` accepts *only* the two given mime types.
-* `{ accept: ['text/*'], reject: ['text/plain'] }` accepts any `text/*` that is not
-  `text/plain`.
+To run a storage server in production you will need to enroll on the network first to
+obtain your provider-id and role account.
 
-More advanced filtering is currently not available.
 
 API Packages
 ------------
@@ -78,7 +72,7 @@ For reusability across API versions, it's best to keep files in the `paths`
 subfolder very thin, and instead inject implementations via the `dependencies`
 configuration value of `express-openapi`.
 
-These implementations line to the `./lib` subfolder. Adjust `server.js` as
+These implementations line to the `./lib` subfolder. Adjust `app.js` as
 needed to make them available to API packages.
 
 Streaming Notes

storage-node/packages/discovery/README.md

@@ -5,25 +5,19 @@ discovery information about themselves, and for consumers to resolve this
 information.
 
 In the Joystream network, services are provided by having members stake for a
-role. The role is identified by a unique actor key. Resolving service information
-associated with the actor key is the main purpose of this module.
+role. The role is identified by a worker id. Resolving service information
+associated with the worker id is the main purpose of this module.
 
 This implementation is based on [IPNS](https://docs.ipfs.io/guides/concepts/ipns/)
 as well as runtime information.
 
 ## Discovery Workflow
 
-The discovery workflow provides an actor public key to the `discover()` function, which
+The discovery workflow provides worker id to the `discover()` function, which
 will eventually return structured data.
 
-Clients can verify that the structured data has been signed by the identifying
-actor. This is normally done automatically, unless a `verify: false` option is
-passed to `discover()`. Then, a separate `verify()` call can be used for
-verification.
-
-Under the hood, `discover()` uses any known participating node in the discovery
-network. If no other nodes are known, the bootstrap nodes from the runtime are
-used.
+Under the hood, `discover()` the bootstrap nodes from the runtime are
+used in a browser environment, or the local ipfs node otherwise.
 
 There is a distinction in the discovery workflow:
 
@@ -31,8 +25,8 @@ There is a distinction in the discovery workflow:
   is performed to discover nodes.
 2. If run in a node.js process, instead:
   - A trusted (local) IPFS node must be configured.
-  - The chain is queried to resolve an actor key to an IPNS peer ID.
-  - The trusted IPFS node is used to resolve the IPNS peer ID to an IPFS
+  - The chain is queried to resolve a worker id to an IPNS id.
+  - The trusted IPFS node is used to resolve the IPNS id to an IPFS
     file.
   - The IPFS file is fetched; this contains the structured data.
 
@@ -45,11 +39,10 @@ The publishing workflow is a little more involved, and requires more interaction
 with the runtime and the trusted IPFS node.
 
 1. A service information file is created.
-1. The file is signed with the actor key (see below).
-1. The file is published on IPFS.
+1. The file is published on IPFS, using the IPNS self key of the local node.
 1. The IPNS name of the trusted IPFS node is updated to refer to the published
    file.
-1. The runtime mapping from the actor ID to the IPNS name is updated.
+1. The runtime mapping from the worker ID to the IPNS name is updated.
 
 ## Published Information
 
@@ -57,10 +50,7 @@ Any JSON data can theoretically be published with this system; however, the
 following structure is currently imposed:
 
 - The JSON must be an Object at the top-level, not an Array.
-- Each key must correspond to a service spec (below).
-
-The data is signed using the [@joystream/json-signing](../json-signing/README.md)
-package.
+- Each key must correspond to a [service spec](../../docs/json-signing/README.md).
 
 ## Service Info Specifications
 

storage-node/packages/helios/README.md

@@ -6,7 +6,6 @@ A basic tool to scan the joystream storage network to get a birds eye view of th
 ## Scanning
 
 ```
-yarn
-yarn run helios
+yarn helios
 ```
 

storage-node/packages/storage/README.md

@@ -2,9 +2,6 @@
 
 This package contains an abstraction over the storage backend of colossus.
 
-Its main purpose is to allow testing the storage subsystem without having to
-run a blockchain node.
-
 In the current version, the storage is backed by IPFS. In order to run tests,
 you have to also run an [IPFS node](https://dist.ipfs.io/#go-ipfs).