# [KPI 36.OP-1](https://blog.joystream.org/sumer-kpis/#36.OP-1) The goal is to install a [GIZA](https://github.com/Joystream/joystream/tree/giza_staging) validator, query node, storage provider and distributor on a fresh VPS. ## Setup ### Node See [Network deployment](../Network-deployment) ### [Query-node](https://github.com/Joystream/joystream/tree/giza_staging/query-node) Edit: Some instructions have been simplified since the test ([README olympia QN](https://github.com/Lezek123/substrate-runtime-joystream/tree/olympia-readme/query-node)). Download these to `~/joystream` (or your repository root): - [.env](.env) - [import-qn-data.sh](import-qn-data.sh) - [deploy-qn.sh](deploy-qn.sh) - [reset-qn.sh](reset-qn.sh) Adapt `JOYSTREAM_NODE_WS` in `.env` and `import-qn-data.sh` if necessary. ``` cd ~/joystream/query-node yarn build nano manifest.yml # change second to last line to: --- height: "[13876,13876]" --- cd .. sudo docker-compose up chmod +x import-qn-data.sh deploy-qn.sh reset-qn.sh ./reset-qn.sh ``` ### Storage See [storage-node](https://github.com/Joystream/joystream/tree/giza_staging/storage-node). ### Distributor See [distributor-node](https://github.com/Joystream/joystream/tree/giza_staging/distributor-node). ### Pioneer ``` cd ~/joystream/pioneer yarn yarn build:www rsync -aP packages/apps/build/ /var/www/pioneer ``` To pre-select your endpoint, edit `packages/apps-config/src/settings/endpoints.js` and add ``` { info: 'Joystream Giza', text: t('giza-l1dev', 'Joystream Giza (l1dev)', { ns: 'apps-config' }), value: 'wss://giza-l1dev.joystream.app/rpc' }, ``` as first array entry for `createLive` (line 23). Run `yarn build:www` again and you should see: > i18next-scanner: Added a new translation key { "giza-l1dev": "Joystream Giza (l1dev)" } to "packages/apps/public/locales/en/apps-config.json" Alternatively it is possible to set `WS_URL` during build time: `WS_URL=wss://myendpoint yarn build:www` ### Atlas ``` git clone htps://github.com/joystream/atlas cd atlas yarn yarn build rsync -aP dist/ /var/www/atlas ``` - Set DNS record for `atlas` subdomain (depends on your registrar) - Create `/etc/nginx/sites-available/atlas` to configure nginx site ``` server { listen 443 ssl; server_name atlas.DOMAIN; root /var/www/atlas; index index.html; include /etc/letsencrypt/options-ssl-nginx.conf; ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; } ``` - Activate site ``` cd /etc/nginx/sites-enabled ln -s ../sites-available/atlas ``` - Request TLS certificate: `certbot run`, select the subdomain and hit `enter`. ### Nginx - Install dependencies: `apt install nginx certbot python3-certbot-nginx` - Create `/etc/nginx/sites-available/joysream` to configure nginx site ``` server { listen 80; return 301 https://$host$request_uri; } server { listen 443 ssl; server_name giza-l1dev.joystream.app; root /var/www/joystream; # build atlas or pioneer and move it there index index.html; location /rpc { proxy_pass http://localhost:9944; proxy_redirect off; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; } # query node location /graphql { rewrite /graphql(/.+) $1 break; proxy_pass http://localhost:8081; proxy_set_header Access-Control-Allow-Methods "GET, PUT, HEAD, OPTIONS"; } location /@apollographql { rewrite (.*) /graphql$1 last; } # storage provider location /storage { rewrite /storage/?(.*) /$1 break; proxy_pass http://localhost:3333; proxy_set_header Access-Control-Allow-Methods "GET, PUT, HEAD, OPTIONS"; proxy_set_header Access-Control-Allow-Headers "GET, PUT, HEAD, OPTIONS"; client_max_body_size 50G; } # distributor location /distributor { rewrite /distributor/?(.*) /$1 break; proxy_pass http://localhost:3334; proxy_set_header Access-Control-Allow-Methods "*"; proxy_set_header Access-Control-Allow-Headers "*"; } include /etc/letsencrypt/options-ssl-nginx.conf; ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; } ``` - Activate site ``` cd /etc/nginx/sites-enabled ln -s ../sites-available/joystream rm default ``` - Request TLS certificate: `certbot run`. Enter email address, accept the TOS to create a certbot account. The rest happens automatically. No need to add a redirect because it is already configured above. Test: - open Pioneer and choose as custom endpoint: `wss://giza-l1dev.joystream.app/rpc` - https://giza-l1dev.joystream.app (if pioneer or atlas are set up) - https://giza-l1dev.joystream.app/graphql - https://giza-l1dev.joystream.app/storage/api/v1/version - https://giza-l1dev.joystream.app/storage/api/v1/state/data-objects - https://giza-l1dev.joystream.app/distributor/ ## Configuration 1. Apply as storage provider and distributor, save files to ~/keys/ 2. Have the leads fill the openings 3. Get invited to a bucket 4. Verify group memberships ``` $ ~/joystream/cli/bin/run working-groups:overview Current Group: storageProviders ___________________ Group lead ___________________ Member id: 2010 Member handle: storage_lead_giza Role account: 5GHPwGqcVjt3KKrEUhkgLJaizRymegx5astt4fekGVmL7F4q ____________________ Members _____________________ Worker id Member id Member handle Stake Earned Role account 3 515 l1dev 10.0000 JOY 0 5DNe6u...ATpk38 2 2012 storage_2 100.0000 JOY 0 5DGQLX...Fai2MW 1 2012 storage_2 100.0000 JOY 0 5H8CPz...hoWPtP 0 2010 storage_lead_giza 100.0000 JOY 0 5GHPwG...mL7F4q $ ~/joystream/cli/bin/run working-groups:overview -g distributors Current Group: distributors ___________________ Group lead ___________________ Member id: 2015 Member handle: dist_lead Role account: 5GJWFApQiMqFjhiLWhxtVAjva7fVy7kYAf8mBgME4rGXX4Vy ____________________ Members _____________________ Worker id Member id Member handle Stake Earned Role account 2 515 l1dev 100.0000 JOY 0 5DCCUr...qjEvK9 1 2016 dist_1 100.0000 JOY 0 5GQZGW...m9z7VA 0 2015 dist_lead 100.0000 JOY 0 5GJWFA...GXX4Vy ``` ### Storage Provider - [Accept invitation](https://github.com/Joystream/joystream/tree/giza_staging/storage-node-v2#storage-node-operatoraccept-invitation) ``` $ ~/joystream/storage-node-v2/bin/run operator:accept-invitation -i 2 -w 3 -k ~/keys/giza-storage-l1dev.json -t 5DNe6ubsQhmcSZRZdBLJroe9qKThW9CsSGRKHiZx8pATpk38 2022-01-04 15:43:26:4326 info: Initialized runtime connection: ws://localhost:9944 2022-01-04 15:43:30:4330 info: Waiting for chain to be synced before proceeding. 2022-01-04 15:43:30:4330 info: Accepting pending storage bucket invitation... 2022-01-04 15:43:30:4330 debug: Sending storage.acceptStorageBucketInvitation extrinsic... 2022-01-04 15:43:36:4336 debug: Extrinsic successful! ``` - set [metadata](https://github.com/Joystream/joystream/tree/giza_staging/storage-node-v2#metadata): [`storage-node operator:set-metadata`](https://github.com/Joystream/joystream/tree/giza_staging/storage-node-v2#storage-node-operatorset-metadata) ([example json](https://github.com/Joystream/joystream/blob/giza_staging/storage-node-v2/scripts/operatorMetadata.json) - [fields](https://github.com/Joystream/joystream/blob/giza_staging/metadata-protobuf/proto/Storage.proto)) ``` $ ~/joystream/storage-node-v2/bin/run operator:set-metadata -i 2 -w 3 -j ~/bucket-2-metadata.json -k ~/keys/giza-storage-l1dev.json 2022-01-04 15:46:21:4621 info: Initialized runtime connection: ws://localhost:9944 2022-01-04 15:46:24:4624 info: Waiting for chain to be synced before proceeding. 2022-01-04 15:46:24:4624 info: Setting the storage operator metadata... 2022-01-04 15:46:24:4624 debug: Sending storage.setStorageOperatorMetadata extrinsic... 2022-01-04 15:46:30:4630 debug: Extrinsic successful! ``` ### Distributor See [docs/node](https://github.com/Joystream/joystream/blob/giza_staging/distributor-node/docs/node/index.md) - Accept invitaiton ``` ~/joystream/distributor-node/bin/run operator:accept-invitation -B 2:0 -w 2 -c ~/joystream/distributor-node/config.yml 2022-01-04 16:08:16:816 CLI info: Accepting distribution bucket operator invitation... { "bucketId": { "distribution_bucket_family_id": "2", "distribution_bucket_index": "0" }, "workerId": 2 } ? Tx fee of 0 will be deduced from you account, do you confirm the transfer? (y/N) y ? Tx fee of 0 will be deduced from you account, do you confirm the transfer? (y/N) y? Tx fee of 0 will be deduced from you account, do you confirm the transfer? Yes 2022-01-04 16:08:22:822 SubstrateApi info: Sending storage.acceptDistributionBucketInvitation extrinsic from 5DCCUrwUZuqBHMMcaswK5rk1DpFNGvuZvoZHWiHYNyqjEvK9 2022-01-04 16:08:24:824 CLI info: Invitation succesfully accepted! ``` - set [metadata](https://github.com/Joystream/joystream/blob/giza_staging/distributor-node/docs/node/index.md#metadata) (or use the [API](https://github.com/Joystream/joystream/blob/giza_staging/distributor-node/docs/api/operator/index.md)) ``` ~/joystream/distributor-node/bin/run operator:set-metadata -B 2:0 -w 2 -c ~/joystream/distributor-node/config.yml -i ~/distributor-metadata.json 2022-01-04 16:25:29:2529 CLI info: Setting bucket operator metadata... { "bucketId": { "distribution_bucket_family_id": "2", "distribution_bucket_index": "0" }, "workerId": 2, "metadata": { "endpoint": "https://giza-l1dev.joystream.app/distributor", "location": { "countryCode": "US-NJ", "city": "Newark", "coordinates": { "latitude": 40, "longitude": 74 } }, "extra": "Welcome to Joystream GIZA - America branch! -- operated by l1dev (dev@joystreamstats.live)" } } ? Tx fee of 0 will be deduced from you account, do you confirm the transfer? (y/N) y ? Tx fee of 0 will be deduced from you account, do you confirm the transfer? (y/N) y? Tx fee of 0 will be deduced from you account, do you confirm the transfer? Yes 2022-01-04 16:25:39:2539 SubstrateApi info: Sending storage.setDistributionOperatorMetadata extrinsic from 5DCCUrwUZuqBHMMcaswK5rk1DpFNGvuZvoZHWiHYNyqjEvK9 2022-01-04 16:25:42:2542 CLI info: Bucket operator metadata succesfully set/updated! ``` ## Runtime ### Staking and Rewards (WGs) > For each group (incl. Leads), check that: > - staking and unstaking works as expected > - rewards are paid out as they should ### Validator Rewards > The runtime upgrade included a change in these params. For each `era`, get: > - total issuance > - total stake > - total reward > - ~blockheight | Block | Era | Issuance | Stake | % Staked | Era Reward | Action | |---|---|---|---|---|---|---| | 0 | 0 | 574460220 | 15000 | 0.0 | 3275 | | | 6000 | 10 | 574460120 | 15000 | 0.0 | 3275 | | | 6600 | 11 | 574455020 | 15000 | 0.0 | 5698 | | | 7200 | 12 | 1000000000 | 200000000 | 20.0 | 58841 | | | 13200 | 22 | 1000000000 | 200000000 | 20.0 | 58841 | | | 13800 | 23 | 999999800 | 200000000 | 20.0 | 14805 | | | 14400 | 24 | 999999700 | 200000000 | 20.0 | 14780 | | | 15000 | 25 | 999999700 | 100007500 | 10.0 | 10250 | | | 15600 | 26 | 999999600 | 100007500 | 10.0 | 10250 | | | 16200 | 27 | 999999600 | 100007500 | 10.0 | 10250 | | | 16800 | 28 | 999999600 | 200000000 | 20.0 | 14805 | | | 151800 | 253 | 999999500 | 200000000 | 20.0 | 14805 | | | 152400 | 254 | 999999500 | 200000000 | 20.0 | 14780 | | | 153000 | 255 | 999999500 | 200000000 | 20.0 | 14805 | | | 161400 | 269 | 999999400 | 200000000 | 20.0 | 14805 | | | 162000 | 270 | 999999400 | 200500000 | 20.1 | 14828 | | | 187800 | 313 | 999999400 | 200500000 | 20.1 | 14828 | | | 188400 | 314 | 999999400 | 200500000 | 20.1 | 15397 | | | 192000 | 320 | 1099991546 | 200500000 | 18.2 | 15397 | | ## Storage The storage tests are delayed to improve the content migration script (for non-sudo uploads and to support dynamic channel-bucket-provider assignment - see ([PR #3034](https://github.com/Joystream/joystream/pull/3034)). 1. What happens when a storage node runs out of (local) storage? - eg. the bucket is configured to accept 100GB, whereas the node runs out at 80GB 2. What happens when a bucket is full (ref. bucket config on-chain) and you upload more from a channel where it's assigned to accept from? 3. What happens when a bucket is full (ref. bucket config on-chain) and you assign a new bag to it? 4. What happens when a SP node, that is the only source of some data goes down? - Cases: 1. No distributor has it (or is assigned): Try to assign the channel bag to another SP, and use the re-upload command (cli and/or atlas) 2. One or more distributors has it (and is assigned): Try to assign the channel bag to another SP, and use the re-upload command (cli and/or atlas) 3. One or more distributors has it (and is assigned): Try to assign the channel bag to another SP, and try to fetch 5it from the distributor 5. How does the blacklist work? - Cases: 1. One distributor has (and is assigned) the file. asset it get censored in atlas? - What happens if you upload the asset again? 2. No distributor has the file (or is assigned). When one gets assigned it, does it get it? - What happens if you upload the asset again? ### CLI > - Check that doing `--help` for all commands > - provides a useful description > - everything is correct > - syntax is consistent See [Commands](Commands.md#storage) ## Distributor 1. What happens when a distributor node runs out of storage? - eg. the bucket is configured to accept 100GB, whereas the node runs out at 80GB ### CLI > - Check that doing `--help` for all commands > - provides a useful description > - everything is correct > - syntax is consistent > - all the examples are correct See [Commands](Commands.md#distributor) No help: - start - dev:batchUpload - node:* - leader:cancel-invitation - leader:create-bucket-family - leader:delete-bucket-family - leader:invite-bucket-operator - leader:remove-bucket-operator - leader:set-bucket-family-metadata - leader:set-buckets-per-bag-limit - leader:update-bucket-mode # Hours | Date | Hours | Task | |-------------|-------|------------------------------------------------| | Jan 2 13-17 | 4 | start node, storage provider, distributor | | Jan 4 15-18 | 3 | storage provider and distributor configuration | | Jan 4 19-22 | 3 | check commands, validator eras | | Jan 5 20-23 | 3 | nginx config | | Jan 6 18-23 | 5 | content creation | | Jan 7 12-15 | 3 | query-node deployment | | Jan 10 6-11 | 5 | research joystream issue updates | | Jan 13 6-12 | 6 | content-migration script, PR #3034 | | Jan 14 2-7 | 5 | content-migration script, PR #3034 | | | 37 | |