Mokhtar Naamani f4cb727dec bump types and metadata npm package version | 2 jaren geleden | |
---|---|---|
.. | ||
bin | 2 jaren geleden | |
scripts | 2 jaren geleden | |
src | 2 jaren geleden | |
.eslintignore | 2 jaren geleden | |
.eslintrc.js | 2 jaren geleden | |
.gitignore | 2 jaren geleden | |
.prettierignore | 2 jaren geleden | |
README.md | 2 jaren geleden | |
package.json | 2 jaren geleden | |
tsconfig.json | 2 jaren geleden |
Joystream storage subsystem.
The main responsibility of Colossus is handling media data for users. The data could be images, audio, or video files. Colossus receives uploads and saves files in the local folder, registers uploads in the blockchain, and later serves files to Argus nodes (distribution nodes). Colossus instances spread the data using peer-to-peer synchronization. Data management is blockchain-based, it relies on the concepts of buckets, bags, data objects. The full description of the blockchain smart contracts could be found here.
Colossus provides REST API for its clients and other Colossus instances. It's based on the OpenAPI Specification v3. Here is the complete spec (work in progress).
API endpoints:
state
data statistics - total data folder size and data object number
There is a command-line interface to manage Storage Working Group operations like create a bucket or change storage settings. Full description could be found below.
There are several groups of command:
server
starts Colossus and help
shows the full command list.The storage provider should provide metadata for Colossus instance to be discoverable by other Colossus or Argus (distributor node) instances. At the very least an endpoint should be registered in the blockchain. For some complex scenarios, Colossus should provide its geolocation.
Metadata could be registered using operator:set-metadata command.
A simple endpoint could be set using the --endpoint
flag of the command. Complex metadata requires JSON file (example).
JSON file format based on the protobuf format described here.
Colossus accepts files using its API. The data must be uploaded using POST http method with multipart/form-data
.
Simplified process (file uploading):
accepted
in the blockchainSeveral instances of Colossus should contain the data replica in order to provide some degree of reliability.
When some Colossus instance receives the data upload it marks the related data object as accepted
.
Other instances that have the same obligations to store the data (they serve storage buckets assigned to the same bag)
will eventually load this data object from the initial receiver (or some other node that already downloaded a new
data object from the initial receiver) using REST API.
The actual data distribution (serving to end-users) is done via Argus - the distributor node. It gets data from Colossus using the same get
endpoint on a single data object basis.
WorkerId
from the Storage Working group.# Ubuntu Linux
# Install packages required for installation
apt update
apt install git curl
# Clone the code repository
git clone https://github.com/Joystream/joystream
cd joystream
# Install volta
curl https://get.volta.sh | bash
bash
# Install project dependencies and build it
yarn
yarn workspace @joystream/types build
yarn workspace @joystream/metadata-protobuf build
yarn workspace storage-node build
# Verify installation
cd storage-node
yarn storage-node version
$ yarn storage-node server --apiUrl ws://localhost:9944 -w 0 --accountUri //Alice -q localhost:8081 -o 3333 -d ~/uploads --sync
Full command description could be find below.
There is also an option to run Colossus as Docker container.
storage-node dev:multihash
storage-node dev:sync
storage-node dev:upload
storage-node dev:verify-bag-id
storage-node help [COMMAND]
storage-node leader:cancel-invite
storage-node leader:create-bucket
storage-node leader:delete-bucket
storage-node leader:invite-operator
storage-node leader:remove-operator
storage-node leader:set-bucket-limits
storage-node leader:set-global-uploading-status
storage-node leader:update-bag
storage-node leader:update-bag-limit
storage-node leader:update-blacklist
storage-node leader:update-bucket-status
storage-node leader:update-data-fee
storage-node leader:update-dynamic-bag-policy
storage-node leader:update-voucher-limits
storage-node operator:accept-invitation
storage-node operator:set-metadata
storage-node server
storage-node dev:multihash
Creates a multihash (blake3) for a file.
USAGE
$ storage-node dev:multihash
OPTIONS
-f, --file=file (required) Path for a hashing file.
-h, --help show CLI help
See code: src/commands/dev/multihash.ts
storage-node dev:sync
Synchronizes the data - it fixes the differences between local data folder and worker ID obligations from the runtime.
USAGE
$ storage-node dev:sync
OPTIONS
-d, --uploads=uploads (required) Data uploading directory (absolute path).
-h, --help show CLI help
-o, --dataSourceOperatorUrl=dataSourceOperatorUrl [default: http://localhost:3333] Storage node url base (e.g.:
http://some.com:3333) to get data from.
-p, --syncWorkersNumber=syncWorkersNumber [default: 20] Sync workers number (max async operations in
progress).
-q, --queryNodeEndpoint=queryNodeEndpoint [default: http://localhost:8081/graphql] Query node endpoint (e.g.:
http://some.com:8081/graphql)
-t, --syncWorkersTimeout=syncWorkersTimeout [default: 30] Asset downloading timeout for the syncronization (in
minutes).
-w, --workerId=workerId (required) Storage node operator worker ID.
See code: src/commands/dev/sync.ts
storage-node dev:upload
Upload data object (development mode only).
USAGE
$ storage-node dev:upload
OPTIONS
-c, --cid=cid (required) Data object IPFS content ID.
-h, --help show CLI help
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-s, --size=size (required) Data object size.
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/dev/upload.ts
storage-node dev:verify-bag-id
The command verifies bag id supported by the storage node. Requires chain connection.
USAGE
$ storage-node dev:verify-bag-id
OPTIONS
-h, --help
show CLI help
-i, --bagId=bagId
(required) Bag ID. Format: {bag_type}:{sub_type}:{id}.
- Bag types: 'static', 'dynamic'
- Sub types: 'static:council', 'static:wg', 'dynamic:member', 'dynamic:channel'
- Id:
- absent for 'static:council'
- working group name for 'static:wg'
- integer for 'dynamic:member' and 'dynamic:channel'
Examples:
- static:council
- static:wg:storage
- dynamic:member:4
See code: src/commands/dev/verify-bag-id.ts
storage-node help [COMMAND]
display help for storage-node
USAGE
$ storage-node help [COMMAND]
ARGUMENTS
COMMAND command to show help for
OPTIONS
--all see all commands in CLI
See code: @oclif/plugin-help
storage-node leader:cancel-invite
Cancel a storage bucket operator invite. Requires storage working group leader permissions.
USAGE
$ storage-node leader:cancel-invite
OPTIONS
-h, --help show CLI help
-i, --bucketId=bucketId (required) Storage bucket ID
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/cancel-invite.ts
storage-node leader:create-bucket
Create new storage bucket. Requires storage working group leader permissions.
USAGE
$ storage-node leader:create-bucket
OPTIONS
-a, --allow Accepts new bags
-h, --help show CLI help
-i, --invited=invited Invited storage operator ID (storage WG worker ID)
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-n, --number=number Storage bucket max total objects number
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-s, --size=size Storage bucket max total objects size
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/create-bucket.ts
storage-node leader:delete-bucket
Delete a storage bucket. Requires storage working group leader permissions.
USAGE
$ storage-node leader:delete-bucket
OPTIONS
-h, --help show CLI help
-i, --bucketId=bucketId (required) Storage bucket ID
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/delete-bucket.ts
storage-node leader:invite-operator
Invite a storage bucket operator. Requires storage working group leader permissions.
USAGE
$ storage-node leader:invite-operator
OPTIONS
-h, --help show CLI help
-i, --bucketId=bucketId (required) Storage bucket ID
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-w, --operatorId=operatorId (required) Storage bucket operator ID (storage group worker ID)
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/invite-operator.ts
storage-node leader:remove-operator
Remove a storage bucket operator. Requires storage working group leader permissions.
USAGE
$ storage-node leader:remove-operator
OPTIONS
-h, --help show CLI help
-i, --bucketId=bucketId (required) Storage bucket ID
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/remove-operator.ts
storage-node leader:set-bucket-limits
Set VoucherObjectsSizeLimit and VoucherObjectsNumberLimit for the storage bucket.
USAGE
$ storage-node leader:set-bucket-limits
OPTIONS
-h, --help show CLI help
-i, --bucketId=bucketId (required) Storage bucket ID
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-o, --objects=objects (required) New 'voucher object number limit' value
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-s, --size=size (required) New 'voucher object size limit' value
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/set-bucket-limits.ts
storage-node leader:set-global-uploading-status
Set global uploading block. Requires storage working group leader permissions.
USAGE
$ storage-node leader:set-global-uploading-status
OPTIONS
-h, --help show CLI help
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-s, --set=(on|off) (required) Sets global uploading block (on/off).
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/set-global-uploading-status.ts
storage-node leader:update-bag
Add/remove a storage bucket from a bag (adds by default).
USAGE
$ storage-node leader:update-bag
OPTIONS
-a, --add=add
[default: ] ID of a bucket to add to bag
-h, --help
show CLI help
-i, --bagId=bagId
(required) Bag ID. Format: {bag_type}:{sub_type}:{id}.
- Bag types: 'static', 'dynamic'
- Sub types: 'static:council', 'static:wg', 'dynamic:member', 'dynamic:channel'
- Id:
- absent for 'static:council'
- working group name for 'static:wg'
- integer for 'dynamic:member' and 'dynamic:channel'
Examples:
- static:council
- static:wg:storage
- dynamic:member:4
-k, --keyFile=keyFile
Key file for the account. Mandatory in non-dev environment.
-m, --dev
Use development mode
-p, --password=password
Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-r, --remove=remove
[default: ] ID of a bucket to remove from bag
-u, --apiUrl=apiUrl
[default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri
Account URI (optional). Has a priority over the keyFile and password flags. Could be overriden by ACCOUNT_URI
environment variable.
See code: src/commands/leader/update-bag.ts
storage-node leader:update-bag-limit
Update StorageBucketsPerBagLimit variable in the Joystream node storage.
USAGE
$ storage-node leader:update-bag-limit
OPTIONS
-h, --help show CLI help
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-l, --limit=limit (required) New StorageBucketsPerBagLimit value
-m, --dev Use development mode
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/update-bag-limit.ts
storage-node leader:update-blacklist
Add/remove a content ID from the blacklist (adds by default).
USAGE
$ storage-node leader:update-blacklist
OPTIONS
-a, --add=add [default: ] Content ID to add
-h, --help show CLI help
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-r, --remove=remove [default: ] Content ID to remove
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/update-blacklist.ts
storage-node leader:update-bucket-status
Update storage bucket status (accepting new bags).
USAGE
$ storage-node leader:update-bucket-status
OPTIONS
-h, --help show CLI help
-i, --bucketId=bucketId (required) Storage bucket ID
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-s, --set=(on|off) (required) Sets 'accepting new bags' parameter for the bucket (on/off).
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/update-bucket-status.ts
storage-node leader:update-data-fee
Update data size fee. Requires storage working group leader permissions.
USAGE
$ storage-node leader:update-data-fee
OPTIONS
-f, --fee=fee (required) New data size fee
-h, --help show CLI help
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/update-data-fee.ts
storage-node leader:update-dynamic-bag-policy
Update number of storage buckets used in the dynamic bag creation policy.
USAGE
$ storage-node leader:update-dynamic-bag-policy
OPTIONS
-h, --help show CLI help
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-n, --number=number (required) New storage buckets number
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-t, --bagType=(Channel|Member) (required) Dynamic bag type (Channel, Member).
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/update-dynamic-bag-policy.ts
storage-node leader:update-voucher-limits
Update VoucherMaxObjectsSizeLimit and VoucherMaxObjectsNumberLimit for the Joystream node storage.
USAGE
$ storage-node leader:update-voucher-limits
OPTIONS
-h, --help show CLI help
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-o, --objects=objects (required) New 'max voucher object number limit' value
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-s, --size=size (required) New 'max voucher object size limit' value
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/leader/update-voucher-limits.ts
storage-node operator:accept-invitation
Accept pending storage bucket invitation.
USAGE
$ storage-node operator:accept-invitation
OPTIONS
-h, --help show CLI help
-i, --bucketId=bucketId (required) Storage bucket ID
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD
environment variable.
-t, --transactorAccountId=transactorAccountId (required) Transactor account ID (public key)
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev
environment.
-w, --workerId=workerId (required) Storage operator worker ID
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password
flags. Could be overriden by ACCOUNT_URI environment variable.
See code: src/commands/operator/accept-invitation.ts
storage-node operator:set-metadata
Set metadata for the storage bucket.
USAGE
$ storage-node operator:set-metadata
OPTIONS
-e, --endpoint=endpoint Root distribution node endpoint
-h, --help show CLI help
-i, --bucketId=bucketId (required) Storage bucket ID
-j, --jsonFile=jsonFile Path to JSON metadata file
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev environment.
-m, --dev Use development mode
-p, --password=password Key file password (optional). Could be overriden by ACCOUNT_PWD environment variable.
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-w, --operatorId=operatorId (required) Storage bucket operator ID (storage group worker ID)
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile and password flags. Could be
overriden by ACCOUNT_URI environment variable.
See code: src/commands/operator/set-metadata.ts
storage-node server
Starts the storage node server.
USAGE
$ storage-node server
OPTIONS
-d, --uploads=uploads (required) Data uploading directory (absolute path).
-e, --elasticSearchEndpoint=elasticSearchEndpoint Elasticsearch endpoint (e.g.: http://some.com:8081).
Log level could be set using the ELASTIC_LOG_LEVEL
enviroment variable.
Supported values: warn, error, debug, info. Default:debug
-h, --help show CLI help
-i, --syncInterval=syncInterval [default: 1] Interval between synchronizations (in
minutes)
-k, --keyFile=keyFile Key file for the account. Mandatory in non-dev
environment.
-l, --logFilePath=logFilePath Absolute path to the rolling log files.
-m, --dev Use development mode
-n, --logMaxFileNumber=logMaxFileNumber [default: 7] Maximum rolling log files number.
-o, --port=port (required) Server port.
-p, --password=password Key file password (optional). Could be overriden by
ACCOUNT_PWD environment variable.
-q, --queryNodeEndpoint=queryNodeEndpoint (required) [default: http://localhost:8081/graphql] Query
node endpoint (e.g.: http://some.com:8081/graphql)
-r, --syncWorkersNumber=syncWorkersNumber [default: 20] Sync workers number (max async operations in
progress).
-s, --sync Enable data synchronization.
-t, --syncWorkersTimeout=syncWorkersTimeout [default: 30] Asset downloading timeout for the
syncronization (in minutes).
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory
in non-dev environment.
-w, --worker=worker (required) Storage provider worker ID
-x, --logMaxFileSize=logMaxFileSize [default: 50000000] Maximum rolling log files size in
bytes.
-y, --accountUri=accountUri Account URI (optional). Has a priority over the keyFile
and password flags. Could be overriden by ACCOUNT_URI
environment variable.
-z, --logFileChangeFrequency=(yearly|monthly|daily|hourly) [default: daily] Log files update frequency.
See code: src/commands/server.ts