Update the storage documentation (#851)

* new structure
* add upgrade
* Node dir
* joystream service node
* Storage node guide
* Distributors
* Metadata
* Setup
* fix links
* Distributor node setup
* Add docker installation
* Update Caddyfile
* add monitoring
* add DP monitoring
* add monitoring geoip
* move leader folder
* Create Strorage_WG_Deputy_Leader.json
* Create Strorage_WG_Worker.json
* Create Strorage_WG_Leader.json.json
* Create Upload Test procedure.md
* Create GraphQL.md
* Create Budget.md
* Create Commands
* tools dir
* Create opening
* Delete opening
* Create mainnet-questions-for-Leads.md
* update storage install
* setup node remove telemetry
* Update README.md

working-groups/storage-group/NodeSteup/README.md

@@ -14,7 +14,7 @@ To upgrade the node please  [go here for the upgrade guide](./Upgrade/README.md)
 ## Hardware
 - CPU: 6 Core
 - RAM: 16G
-- Storage: 2T SSD
+- Storage: 10T SSD
 - Bandwidth: 1G
 
 ## Location

working-groups/storage-group/NodeSteup/joystream-node/README.md

@@ -12,12 +12,12 @@ $ cd ~/
 $ mkdir joystream-node
 $ cd joystream-node
 # 64 bit debian based Linux
-$ wget https://github.com/Joystream/joystream/releases/download/v10.7.1/joystream-node-6.7.0-bdec855-x86_64-linux-gnu.tar.gz
-$ tar -vxf joystream-node-6.7.0-bdec855-x86_64-linux-gnu.tar.gz
+$ wget https://github.com/Joystream/joystream/releases/download/v11.3.0/joystream-node-7.4.1-d2243721017-x86_64-linux-gnu.tar.gz
+$ tar -vxf joystream-node-7.4.1-d2243721017-x86_64-linux-gnu.tar.gz
 $ mv joystream-node /usr/local/bin/
-$ wget https://github.com/Joystream/joystream/releases/download/v10.5.0/joy-testnet-6.json
+$ wget https://github.com/Joystream/joystream/releases/download/v11.3.0/joy-testnet-7-carthage.json
 # Test is it working. 
-$ joystream-node --chain joy-testnet-6.json --pruning archive --validator
+$ joystream-node --chain joy-testnet-7-carthage.json --pruning archive --validator
 ```
 - If you want your node to have a non-random identifier, add the flag:
   - `--name <nodename>`
@@ -78,11 +78,11 @@ Type=simple
 User=joystream
 WorkingDirectory=/<path to work directory>/joystream-node/
 ExecStart=joystream-node \
-        --chain /<path to work directory>/joystream-node/joy-testnet-6.json \
+        --chain /<path to work directory>/joystream-node/joy-testnet-7-carthage.json \
         --pruning archive \
         --validator \
         --name <memberId-memberHandle> \
-        --log runtime,txpool,transaction-pool,trace=sync
+        --rpc-cors all
 Restart=on-failure
 RestartSec=3
 LimitNOFILE=10000
@@ -104,13 +104,13 @@ After=network.target
 [Service]
 Type=simple
 User=root
-WorkingDirectory=/<path to work directory>/joystream-node/
+WorkingDirectory=/root/joystream-node/joystream-node/
 ExecStart=joystream-node \
-        --chain /<path to work directory>/joystream-node/joy-testnet-6.json \
+        --chain /root/joystream-node/joy-testnet-7-carthage.json \
         --pruning archive \
         --validator \
-        --name <memberId-memberHandle> \
-        --log runtime,txpool,transaction-pool,trace=sync
+        --name YourCoolName \
+        --rpc-cors all
 Restart=on-failure
 RestartSec=3
 LimitNOFILE=10000

working-groups/storage-group/leader/Commands.md

@@ -38,6 +38,11 @@ yarn storage-node leader:remove-operator -i 8 -k /root/keys/storage-role-key.jso
 yarn storage-node leader:delete-bucket -i 8 -k /root/keys/storage-role-key.json -p xxxxx
 ```
 
+## Re-invite
+```
+yarn storage-node leader:remove-operator -i 3 -k /root/keys/storage-role-key.json -p xxxxxx
+yarn storage-node leader:invite-operator -i 3  -w 3 -k /root/keys/storage-role-key.json -p xxxxx
+```
 ## Evict worker 
 Make sure the bucket is empty and deleted
 ```

working-groups/storage-group/leader/Initial_ setup_commands.md

@@ -24,17 +24,28 @@ yarn storage-node leader:update-bag-limit -l 10 -k /root/keys/storage-role-key.j
 
 ### hiring
 ```
-yarn joystream-cli working-groups:createOpening -o ~/joystream_working_dir/Strorage_WG_Worker.json
+
+yarn joystream-cli working-groups:createOpening -i /root/community-repo/working-groups/storage-group/leader/opening/Strorage_WG_Deputy_Leader.json
+yarn joystream-cli working-groups:createOpening -i /root/community-repo/working-groups/storage-group/leader/opening/Strorage_WG_Worker.json
+
+#list all opening
 yarn joystream-cli working-groups:openings
-yarn joystream-cli working-groups:opening --id 1
-yarn joystream-cli working-groups:application 2
-yarn joystream-cli working-groups:application 3
-yarn joystream-cli working-groups:fillOpening --openingId 1 --applicationIds 2 --applicationIds 3
-yarn joystream-cli working-groups:overview
-yarn joystream-cli working-groups:createOpening -i ~/joystream_working_dir/Strorage_WG_Worker.json
+
+#list an opening
+yarn joystream-cli working-groups:opening --id x
+
+#view application
+yarn joystream-cli working-groups:application x
+
+#Accept application
+yarn joystream-cli working-groups:fillOpening --openingId x --applicationIds x
+
 ```
 ### bucket mgmt
 ```
-yarn storage-node leader:create-bucket -i 18 -n 20000 -s 1500000000000 -k /root/keys/storage-role-key.json -p xxxxxx
-yarn storage-node leader:update-bucket-status -i 18 -s off -k /root/keys/storage-role-key.json -p xxxxxx
+yarn storage-node leader:create-bucket -i <worker_ID> -n 20000 -s 2000000000000 -k /root/keys/storage-role-key.json -p xxxxxx
+yarn storage-node leader:update-bucket-status -i <worker_ID> -s off -k /root/keys/storage-role-key.json -p xxxxxx
 ```
+
+
+yarn joystream-cli working-groups:overview

working-groups/storage-group/leader/mainnet/mainnet-questions-for-Leads.md

@@ -31,7 +31,7 @@ See the job descriptions above.
 
 This should be purely based on skills and abilities including:
  - Linux and devops skill level
- - Machine specification
+ - Infrastructure specification
  - Willing to commit time and expertise to develop tools and procedures i.e. addd value. 
 
 Preference will be made for testnet contributors who showed the above skills, reliability and commitment.

working-groups/storage-group/leader/opening/Strorage_WG_ES_Worker.json

@@ -0,0 +1,58 @@
+{
+    "applicationDetails": "Storage WG ElasticSearch Worker",
+    "hiringLimit": 1,
+    "title": "Storage WG ElasticSearch Worker needed",
+    "shortDescription": "Joystream Video DAO (https://joystream.org) is actively looking for a Storage WG ElasticSearch Worker",
+    "description": "[Joystream Video DAO](https://joystream.org) is actively looking for a Storage WG ElasticSearch Worker to join our [Pioneer Governance App](https://dao.joystream.org) community Storage team.\n\n**Tasks for Carthage:** \n\n- See https://gist.github.com/bwhm/2a8733fe35974d4cb90a34ab1916d2ca#storage-and-distribution\n\n**Responsibilities:** \n\n- Operate a ElasticSearch server and ensure high availability.\n\n- Work with other workgroup members to setup their ElasticSearch clients and make sure the server is collecting their data. - Setup monitoring templates and dashboards to give visibility into the health of the infrastructure **Requirements:** \n\n- ElasticSearch experience is preferred\n\n- High availability: First few days after hired will require high availability and lots of work\n\n- Experience with how to setup and maintain blockchain and ElasticSearch nodes\n\n- Having access to highly performant and reliable IT infrastructure (dedicated servers) with storage capacity of 4TB and more\n\n- Skills: Linux, devops, bash, Docker/Docker-compose, nginx or Caddy, GraphQl, ElasticSearch, (nice to have prometheus/Grafana)\n\n**Rules for firings:** \n\n- https://github.com/Joystream/community-repo/tree/master/working-groups/storage-group/SOP\n\n**About WG:** \n\nNotion page: https://www.notion.so/joystream/Storage-9dc5a16444934dc4bda08b596bc15375\n\nDiscord channel: https://discord.gg/fFRdNHsW\n\n**Stake:** \n\n- $5000 (83,333 JOY)\n\n**Salary:** \n\n- $500 (8,333 JOY)/ month\n\n- If an employee works fewer than 160 hours per month, the Council may reduce their pay according to the hourly rate and actual number of hours worked",
+    "applicationFormQuestions": [{
+            "question": "Are you currently employed?"
+        },
+        {
+            "question": "What is your timezone?"
+        },
+        {
+            "question": "Are you based in US?"
+        },
+        {
+            "question": "Do you have experience with Linux? If yes, please describe it"
+        },
+        {
+            "question": "Do you have experience with Docker? If yes, please describe it"
+        },
+        {
+            "question": "Do you have experience with devops and automation? If yes, please describe it"
+        },
+        {
+            "question": "Do you have experience with GraphQl? If yes, please describe it"
+        },
+        {
+            "question": "Do you have experience with blockchain nodes? If yes, please describe it"
+        },
+	{
+            "question": "Do you have experience with ElasticSearch server? If yes, please describe it"
+        },
+        {
+            "question": "Tell us a bit about youself"
+        },
+        {
+            "question": "Your availability, in hours per day"
+        },
+        {
+            "question": "Do you understand the compensation model?"
+        },
+        {
+            "question": "Tell us a bit more about your experience in IT"
+        },
+	{
+            "question": "Please provide infrastructure specification and location"
+        },
+        {
+            "question": "Your Discord"
+        }
+    ],
+    "stakingPolicy": {
+        "amount": 833330000000000,
+        "unstakingPeriod": 604800
+    },
+    "rewardPerBlock": 192700000
+}

working-groups/storage-group/leader/opening/Strorage_WG_Worker.json

@@ -1,17 +1,9 @@
 {
     "applicationDetails": "Storage WG Worker",
+    "hiringLimit": 5,
     "title": "Storage WG Worker needed",
     "shortDescription": "Joystream Video DAO (https://joystream.org) is actively looking for a Storage WG Worker",
-    "description": "[Joystream Video DAO](https://joystream.org) is actively looking for a Storage WG Worker to join our [Pioneer Governance App](https://dao.joystream.org) community Storage team.\n\n
-    **Who are we:**\n\n
-    - Joystream is an open-source blockchain project that will be truly user-governed. \n\n 
-    - [Storage](https://www.notion.so/joystream/Storage-9dc5a16444934dc4bda08b596bc15375) are a small groups of node maintainers helping to make Jsgenesis products works smoothly and reliably\n\n
-    **Who are you:** \n\n
-    - Experienced with how to setup and maintain blockchain nodes \n\n
-    - Have access to highly performant and reliable IT infrastructure (dedicated servers) with storage capacity of 2TB and more \n\n 
-    - Comply to the group standard operation procedures. \n\n
-    - Skills: Linux, devops, bash, Docker/Docker-compose, nginx or Caddy, GraphQl, (nice to have prometheus/Grafana).
-In order to apply for the opening, every applicant should make a test task (bounty) https://www.notion.so/joystream/Bounty-Storage-WG-Entry-Level-74b13f81a32d4efb811f8259d6fbeee0 ",
+    "description": "[Joystream Video DAO](https://joystream.org) is actively looking for a Storage WG Worker to join our [Pioneer Governance App](https://dao.joystream.org) community Storage team.\n\n**Tasks for Carthage:** \n\n- See https://gist.github.com/bwhm/2a8733fe35974d4cb90a34ab1916d2ca#storage-and-distribution\n\n**Responsibilities:** \n\n- Operate a storage node and ensure high availability.\n\n**Requirements:** \n\n- Storage Worker experience is preferred\n\n- High availability: First few days after hired will require high availability and lots of work\n\n- Experience with how to setup and maintain blockchain nodes\n\n- Having access to highly performant and reliable IT infrastructure (dedicated servers) with storage capacity of 10TB and more\n\n- Skills: Linux, devops, bash, Docker/Docker-compose, nginx or Caddy, GraphQl, (nice to have prometheus/Grafana)\n\n**Rules for firings:** \n\n- https://github.com/Joystream/community-repo/tree/master/working-groups/storage-group/SOP\n\n**About WG:** \n\nNotion page: https://www.notion.so/joystream/Storage-9dc5a16444934dc4bda08b596bc15375\n\nDiscord channel: https://discord.gg/fFRdNHsW\n\n**Stake:** \n\n- $5000 (83,333 JOY)\n\n**Salary:** \n\n- $500 (8,333 JOY)/ month\n\n- If an employee works fewer than 160 hours per month, the Council may reduce their pay according to the hourly rate and actual number of hours worked",
     "applicationFormQuestions": [{
             "question": "Are you currently employed?"
         },
@@ -48,13 +40,16 @@ In order to apply for the opening, every applicant should make a test task (boun
         {
             "question": "Tell us a bit more about your experience in IT"
         },
+	{
+            "question": "Please provide infrastructure specification and location"
+        },
         {
             "question": "Your Discord"
         }
     ],
     "stakingPolicy": {
-        "amount": 100000,
-        "unstakingPeriod": 43201
+        "amount": 833330000000000,
+        "unstakingPeriod": 604800
     },
-    "rewardPerBlock": 2
+    "rewardPerBlock": 23116040
 }

working-groups/storage-group/leader/opening/backup/Strorage_WG_Deputy_Leader.json

@@ -0,0 +1,61 @@
+{
+    "applicationDetails": "Storage WG Deputy Lead",
+    "title": "Storage WG Deputy Lead needed",
+    "shortDescription": "Joystream Video DAO (https://joystream.org) is actively looking for a Storage WG Deputy Lead",
+    "description": "[Joystream Video DAO](https://joystream.org) is actively looking for a Storage WG Deputy Lead to join our [Pioneer Governance App](https://dao.joystream.org) community Storage team.\n\n 
+    **Deputy Lead should be willing to learn how to manage Lead's responsibilities, including::** \n\n
+    - Maximize the Storage WG scores \n\n 
+    - Ensure that Storage Providers are performing adequately \n\n 
+    - Analyze current storage capacities utilized within the Storage WG and develop steps and processes to improve and expand upon them \n\n  
+    - Prepares weekly reports/plans/summary in a format approved by Council and Jsgenesis",
+    **Who are you:** \n\n
+    - Experience in amanging team and help the team to reach it's objectives.\n\n
+    - Experienced with how to setup and maintain blockchain nodes \n\n
+    - Have access to highly performant and reliable IT infrastructure (dedicated servers) with storage capacity of 2TB and more \n\n 
+    - Skills: Linux, devops, bash, Docker/Docker-compose, nginx or Caddy, GraphQl, (nice to have prometheus/Grafana).
+    "applicationFormQuestions": [{
+            "question": "Are you currently employed?"
+        },
+        {
+            "question": "What is your timezone?"
+        },
+        {
+            "question": "Are you based in US?"
+        },
+        {
+            "question": "Do you have experience with Linux? If yes, please describe it"
+        },
+        {
+            "question": "Do you have experience with Docker? If yes, please describe it"
+        },
+        {
+            "question": "Do you have experience with devops and automation? If yes, please describe it"
+        },
+        {
+            "question": "Do you have experience with GraphQl? If yes, please describe it"
+        },
+        {
+            "question": "Do you have experience with blockchain nodes? If yes, please describe it"
+        },
+        {
+            "question": "Tell us a bit about yourself"
+        },
+        {
+            "question": "Your availability, in hours per day"
+        },
+        {
+            "question": "Do you understand the compensation model?"
+        },
+        {
+            "question": "Tell us a bit more about your experience in IT"
+        },
+        {
+            "question": "Your Discord"
+        }
+    ],
+    "stakingPolicy": {
+        "amount": 500000,
+        "unstakingPeriod": 43201
+    },
+    "rewardPerBlock": 4
+}

working-groups/storage-group/leader/opening/backup/Strorage_WG_Worker.json

@@ -0,0 +1,60 @@
+{
+    "applicationDetails": "Storage WG Worker",
+    "title": "Storage WG Worker needed",
+    "shortDescription": "Joystream Video DAO (https://joystream.org) is actively looking for a Storage WG Worker",
+    "description": "[Joystream Video DAO](https://joystream.org) is actively looking for a Storage WG Worker to join our [Pioneer Governance App](https://dao.joystream.org) community Storage team.\n\n
+    **Who are we:**\n\n
+    - Joystream is an open-source blockchain project that will be truly user-governed. \n\n 
+    - [Storage](https://www.notion.so/joystream/Storage-9dc5a16444934dc4bda08b596bc15375) are a small groups of node maintainers helping to make Jsgenesis products works smoothly and reliably\n\n
+    **Who are you:** \n\n
+    - Experienced with how to setup and maintain blockchain nodes \n\n
+    - Have access to highly performant and reliable IT infrastructure (dedicated servers) with storage capacity of 2TB and more \n\n 
+    - Comply to the group standard operation procedures. \n\n
+    - Skills: Linux, devops, bash, Docker/Docker-compose, nginx or Caddy, GraphQl, (nice to have prometheus/Grafana).
+In order to apply for the opening, every applicant should make a test task (bounty) https://www.notion.so/joystream/Bounty-Storage-WG-Entry-Level-74b13f81a32d4efb811f8259d6fbeee0 ",
+    "applicationFormQuestions": [{
+            "question": "Are you currently employed?"
+        },
+        {
+            "question": "What is your timezone?"
+        },
+        {
+            "question": "Are you based in US?"
+        },
+        {
+            "question": "Do you have experience with Linux? If yes, please describe it"
+        },
+        {
+            "question": "Do you have experience with Docker? If yes, please describe it"
+        },
+        {
+            "question": "Do you have experience with devops and automation? If yes, please describe it"
+        },
+        {
+            "question": "Do you have experience with GraphQl? If yes, please describe it"
+        },
+        {
+            "question": "Do you have experience with blockchain nodes? If yes, please describe it"
+        },
+        {
+            "question": "Tell us a bit about youself"
+        },
+        {
+            "question": "Your availability, in hours per day"
+        },
+        {
+            "question": "Do you understand the compensation model?"
+        },
+        {
+            "question": "Tell us a bit more about your experience in IT"
+        },
+        {
+            "question": "Your Discord"
+        }
+    ],
+    "stakingPolicy": {
+        "amount": 100000,
+        "unstakingPeriod": 43201
+    },
+    "rewardPerBlock": 2
+}