Merge pull request #242 from singulart/new-branch-name

Backend implementation of the Community Bounty 19 (Validator Tool)

community-contributions/validator-report-backend/.gitignore

@@ -0,0 +1,5 @@
+./node_modules/
+./lib/
+package-lock.json
+
+node_modules

community-contributions/validator-report-backend/README.md

@@ -0,0 +1,153 @@
+# Validator Stats Builder
+
+A backend tool which allows Substrate node-runners to obtain an informative timely report about their validator activity. It works off the PostgresSQL database, where blockchain data about blocks, eras and Substrate events is imported. 
+Basic usage: 
+
+```
+curl http://localhost:3000/validator-report?addr=5EhDdcWm4TdqKp1ew1PqtSpoAELmjbZZLm5E34aFoVYkXdRW&start_block=100000&end_block=2000000
+```
+
+Search by date interval is also supported:
+```
+curl http://localhost:3000/validator-report?addr=5EhDdcWm4TdqKp1ew1PqtSpoAELmjbZZLm5E34aFoVYkXdRW&start_time=2021-07-18&end_block=2021-08-31
+```
+In the latter case, the search is performed from July 18 00:00:00  to Aug 31 23:59:59. 
+
+
+## Report Structure Explained
+
+A validator report is produced in a JSON format. 
+ 
+ ## Setup
+ 
+ ### Prerequisites
+ 
+ 1. PostgreSQL database[https://www.postgresql.org/]. The tool was developed and tested against PostgreSQL 13.3 and 12.7. Compatibility with other versions is likely, though not confirmed.
+ 2. NodeJS. Verified versions are 16.4.1 and 15.11.10.
+ 3. Fully synchronized Substrate https://substrate.dev/ node exposing the Websocket endpoint to connect to. We highly recommend connecting to such a node on localhost, for performance reasons. Although, remote nodes should wor, too, the data import will just be very slow. This tool was specifically developed for Joystream https://joystream.org blockchain. Compatibility with other Substrate-based blockchains is not confirmed.
+ 4. Empty database created in PostgreSQL.
+
+
+Clone the repo, ```cd``` to the project folder and execute the build command:
+ ```
+ yarn && yarn build
+ ```
+
+ ## Usage when running from scratch
+
+Run the schema migration (this step will create all needed tables and indices) ``` NODE_ENV=<database name goes here> node lib/init_db.js ```
+
+Run the server by executing ``` NODE_ENV=<database name goes here> node lib/index.js ```
+By default, this will start the application on port 3000 connecting to the localhost Substrate node. To change this behavior, use PORT and RPC_ENDPOINT environment variables, respectively: ``` PORT=5555 RPC_ENDPOINT=wss://joystreamstats.live:9945 NODE_ENV=<database name goes here> node lib/index.js ```
+
+From this point on, the application would capture all the new blocks and events that appear in a blockchain aand store them in a database. But what about the past (historical) blocks? To import those, a separate script needs to be used: ```NODE_ENV=<DB> START_BLOCK=<HEIGHT OF THE FIRST BLOCK TO IMPORT> END_BLOCK=<HEIGHT OF THE LAST BLOCK TO IMPORT> node lib/block_range_import.js```. Block heights are just numbers, so by specifying START_BLOCK=1 and END_BLOCK=10000, you essentially import all blocks from 1 to 10000. 
+
+Depending on the size of your blockchain and your system hardware, the importing of all historical blocks may take from couple of hours to a several days. To speed up the process, we recommended to split the range of blocks you want to import to chunks and import them simultaneously in parallel by running the above script several times. 
+
+Note. ```block_range_import.js``` will not automatically stop after importing the last block. You would need to stop it manually using Ctrl+C. 
+
+#### Example
+For example, if your chain has 2.000.000 blocks, it's wise to split them in chunks by 500.000 and run the script four times like this:
+
+In Terminal window 1:
+```
+NODE_ENV=<DB> START_BLOCK=1 END_BLOCK=500000 node lib/block_range_import.js
+```
+
+In Terminal window 2:
+```
+NODE_ENV=<DB> START_BLOCK=500001 END_BLOCK=1000000 node lib/block_range_import.js
+```
+
+In Terminal window 3:
+```
+NODE_ENV=<DB> START_BLOCK=1000001 END_BLOCK=1500000 node lib/block_range_import.js
+```
+
+In Terminal window 4:
+```
+NODE_ENV=<DB> START_BLOCK=1500001 END_BLOCK=2000000 node lib/block_range_import.js
+```
+
+### Making sure all blocks are imported
+Blockchains are ever-growing systems, constantly producing more and more data, so making sure your database is fully in sync with the chain state and no blocks are missing is very important. 
+First of all, you need to make sure all historical blocks are imported. Log in to your database and execute the following SQL: ```select block from start_blocks;``` This should give you the very first block number that your application imported after the start. So, when importing your historical blocks, you can use this value as an END_BLOCK:
+```
+NODE_ENV=<DB> START_BLOCK=1 END_BLOCK=<'VALUE PRODUCED BY THE SQL'> node lib/block_range_import.js
+```
+
+## Usage when the database dump is available
+Importing blockchain data into a database from scratch is a time-consuming process, so there is another way to bootstrap things. For instance, you want to do the import on local machine because it's fast, but your production database is elsewhere.  
+
+TODO finish this section
+
+## Queries
+
+List of eras where validator was active
+ ```
+select a.key, "eraId", stake_total, stake_own, points, rewards, commission from validator_stats vs inner join accounts a on a.id = vs."accountId" where a.key = '55555555555555555555555555555555555555555555' order by "eraId";
+
+ ```
+
+
+
+Main report to be executed by an endpoint
+
+```
+select 
+	vs."eraId", 
+	stake_total, 
+	stake_own, 
+	points, 
+	rewards, 
+	commission, 
+	subq2.blocks_cnt 
+from 
+	validator_stats vs 
+inner join 
+	accounts a on a.id = vs."accountId" 
+inner join 
+	(select 
+		"eraId", count(b.id) blocks_cnt 
+	from 
+		eras e 
+	join 
+		blocks b 
+	on 
+		b."eraId" = e.id 
+	inner join 
+		accounts a 
+	on 
+		a.id = b."validatorId" 
+	and 
+		b."validatorId" = (select id from accounts where key = '55555555555555555555555555555555555555555555') and e.id = "eraId" group by "eraId") subq2 
+	on 
+		subq2."eraId" = vs."eraId" 
+where 
+	a.key = '55555555555555555555555555555555555555555555' 
+and 
+	vs."eraId" 
+in 
+	(select subq.era from (select distinct("eraId") era, min(id) start_height, min(timestamp) start_time, max(id) end_height, max(timestamp) end_time, (max(id) - min(id)) as era_blocks from blocks where blocks.id > 1 and blocks.id < 2000000 group by blocks."eraId") subq) 
+order by "eraId";
+```
+
+Eras starts and ends (blocks and time)
+```
+select distinct("eraId") era, min(id) start_height, min(timestamp) start_time, max(id) end_height, max(timestamp) end_time, (max(id) - min(id)) as era_blocks from blocks group by blocks."eraId";
+ ```
+
+Ordered list of blocks count produced by validators, per era
+```
+select distinct(e.id) era, a.key account, count(b.id) blocks_cnt from eras e join blocks b on b."eraId" = e.id inner join accounts a on a.id = b."validatorId" group by e.id, account order by blocks_cnt desc;
+```
+
+Same as above, but for one validator
+```
+select distinct(e.id) era, a.key account, count(b.id) blocks_cnt from eras e join blocks b on b."eraId" = e.id inner join accounts a on a.id = b."validatorId" where a.key = '44444444444444444444444444444' group by e.id, account order by blocks_cnt desc;
+```
+
+Find missing blocks (not imported for any reason)
+```
+SELECT generate_series(1, 2000000) except (select id from blocks where id between 1 AND 2000000);
+```

community-contributions/validator-report-backend/package.json

@@ -0,0 +1,34 @@
+{
+  "name": "validator-info-extractor-example",
+  "version": "0.1.0",
+  "main": "lib/index.js",
+  "license": "MIT",
+  "scripts": {
+    "build": "tsc --build tsconfig.json",
+    "extracter": "node lib/index.js"
+  },
+  "dependencies": {
+    "@joystream/types": "^0.16.1",
+    "@polkadot/api": "4.2.1",
+    "@polkadot/api-contract": "4.2.1",
+    "@polkadot/keyring": "^6.0.5",
+    "@polkadot/types": "4.2.1",
+    "@polkadot/util": "^6.0.5",
+    "@polkadot/util-crypto": "^6.0.5",
+    "@polkadot/wasm-crypto": "^4.0.2",
+    "@types/bluebird": "^3.5.35",
+    "@types/bn.js": "^4.11.6",
+    "bn.js": "^5.1.2",
+    "sequelize": "^6.6.5",
+    "axios": "^0.21.1",
+    "chalk": "^4.1.0",
+    "pg": "^8.5.1",
+    "socket.io": "^2.2.0",
+    "express": "^4.17.1",
+    "cors": "^2.8.5"
+  },
+  "devDependencies": {
+    "@polkadot/ts": "^0.3.62",
+    "typescript": "^3.9.7"
+  }
+}

community-contributions/validator-report-backend/src/ascii.ts

@@ -0,0 +1,12 @@
+const jsstats = String.raw`
+       _                 _                             _____ _        _
+      | |               | |                           / ____| |      | | APIv1     
+      | | ___  _   _ ___| |_ _ __ ___  __ _ _ __ ___ | (___ | |_ __ _| |_ ___ 
+  _   | |/ _ \| | | / __| __| '__/ _ \/ _  | '_   _ \ \___ \| __/ _  | __/ __|
+ | |__| | (_) | |_| \__ \ |_| | |  __/ (_| | | | | | |____) | || (_| | |_\__ \
+  \____/ \___/ \__, |___/\__|_|  \___|\__,_|_| |_| |_|_____/ \__\__,_|\__|___/
+                __/ |                                                         
+               |___/ .live/api
+`
+
+export default jsstats

community-contributions/validator-report-backend/src/block_range_import.ts

@@ -0,0 +1,28 @@
+import { addBlockRange, processNext } from './joystream';
+import { connectUpstream } from './joystream/ws';
+import {findLastProcessedBlockId} from './db/models/block'
+import {
+    StartBlock
+} from './db/models'
+
+async function main () {
+
+    const api = await connectUpstream();
+
+    const firstBlock:number = parseInt(process.env.START_BLOCK);
+    const lastBlock:number = parseInt(process.env.END_BLOCK) ||  (await StartBlock.findAll())[0].get({plain: true}).block - 1;
+
+    console.log(`[Joystream] Importing block range [${firstBlock} - ${lastBlock}] started`);
+    
+    const lastImportedBlockHeight = await findLastProcessedBlockId(firstBlock, lastBlock);
+    if (lastImportedBlockHeight && lastImportedBlockHeight > 0 && lastImportedBlockHeight < lastBlock) {
+        console.log(`[Joystream] Found last imported block ${lastImportedBlockHeight}. Resuming processing from the next one`);
+        await addBlockRange(api, lastImportedBlockHeight + 1, lastBlock);
+    } else {
+        await addBlockRange(api, firstBlock, lastBlock);
+    }
+    
+    processNext();
+}  
+
+main()

community-contributions/validator-report-backend/src/db/db.ts

@@ -0,0 +1,9 @@
+import { Sequelize} from 'sequelize'
+
+const dbName = process.env.NODE_ENV;
+const dbUrl = process.env.DATABASE_URL || `postgres://localhost:5432/${dbName}`;
+const db = new Sequelize(dbUrl, {logging: () => false});
+// const db = new Sequelize(dbUrl, {logging: console.log});
+
+// export default new Sequelize(dbUrl, {logging: console.log})
+export default db;

community-contributions/validator-report-backend/src/db/index.ts

@@ -0,0 +1,6 @@
+import db from './db'
+
+require('./models')
+
+
+export default db

community-contributions/validator-report-backend/src/db/models/account.ts

@@ -0,0 +1,18 @@
+import db from '../db'
+
+import { DataTypes, Model } from 'sequelize'
+
+class Account extends Model {}
+
+Account.init({
+  id: {
+    type: DataTypes.INTEGER,
+    autoIncrement: true,
+    primaryKey: true,
+  },
+  key: DataTypes.STRING,
+  format: DataTypes.STRING,
+  about: DataTypes.TEXT,
+}, {modelName: 'account', sequelize: db})
+
+export default Account

community-contributions/validator-report-backend/src/db/models/balance.ts

@@ -0,0 +1,12 @@
+import db from '../db'
+import { DataTypes, Model } from 'sequelize'
+
+class Balance extends Model {}
+
+Balance.init({
+  available: DataTypes.INTEGER,  
+  locked: DataTypes.INTEGER,
+  frozen: DataTypes.INTEGER,
+}, { modelName: 'balance', sequelize: db })
+
+export default Balance

community-contributions/validator-report-backend/src/db/models/block.ts

@@ -0,0 +1,33 @@
+import db from '../db'
+import { DataTypes, Op, Model } from 'sequelize'
+
+class Block extends Model {}
+
+Block.init({
+  id: {
+    type: DataTypes.BIGINT,
+    primaryKey: true,
+  },
+  validatorId: {type: DataTypes.INTEGER, allowNull: false},
+  eraId: {type: DataTypes.INTEGER, allowNull: false},
+  hash: DataTypes.STRING,
+  timestamp: DataTypes.DATE,
+  blocktime: DataTypes.BIGINT,
+}, {modelName: 'block', sequelize: db})
+
+
+export const findLastProcessedBlockId = (start: number, end: number): Promise<number> => {
+  return Block.max('id', 
+    { 
+      where: {
+        id: {
+          [Op.and]: {
+            [Op.gte]: start,
+            [Op.lt]: end,
+          }
+        }
+      }
+  })
+}
+
+export default Block

community-contributions/validator-report-backend/src/db/models/era.ts

@@ -0,0 +1,19 @@
+import db from '../db'
+import { DataTypes, Model } from 'sequelize'
+
+class Era extends Model {}
+
+Era.init({
+  id: {
+    type: DataTypes.INTEGER,
+    primaryKey: true,
+  },
+  waitingValidators: DataTypes.INTEGER,
+  allValidators: DataTypes.INTEGER,
+  timestamp: DataTypes.DATE,
+  stake: DataTypes.DECIMAL,
+  eraPoints: DataTypes.DECIMAL,
+  nominators: DataTypes.INTEGER,
+}, { modelName: 'era', sequelize: db })
+
+export default Era

community-contributions/validator-report-backend/src/db/models/event.ts

@@ -0,0 +1,13 @@
+import db from '../db'
+import { DataTypes, Model } from 'sequelize'
+
+class Event extends Model {}
+
+Event.init({
+  blockId: DataTypes.INTEGER,
+  section: DataTypes.STRING,
+  method: DataTypes.STRING,
+  data: DataTypes.JSONB
+}, { modelName: 'event', sequelize: db })
+
+export default Event

community-contributions/validator-report-backend/src/db/models/index.ts

@@ -0,0 +1,34 @@
+import Account from './account'
+import Balance from './balance'
+import Block from './block'
+import Era from './era'
+import Event from './event'
+import ValidatorStats from './validatorstats'
+import StartBlock from './startblock'
+
+Account.hasMany(Balance)
+
+Balance.belongsTo(Account)
+Balance.belongsTo(Era)
+
+Era.hasMany(Balance)
+Era.hasMany(Block)
+
+Block.belongsTo(Account, { as: 'validator' })
+Block.belongsTo(Era)
+Block.hasMany(Event)
+
+Event.belongsTo(Block)
+
+ValidatorStats.belongsTo(Era)
+ValidatorStats.belongsTo(Account)
+
+export {
+  Account,
+  Balance,
+  Block,
+  Era,
+  Event,
+  ValidatorStats,
+  StartBlock
+}

community-contributions/validator-report-backend/src/db/models/startblock.ts

@@ -0,0 +1,10 @@
+import db from '../db'
+import { DataTypes, Model } from 'sequelize'
+
+class StartBlock extends Model {}
+
+StartBlock.init({
+  block: DataTypes.INTEGER,
+}, {modelName: 'start_block', sequelize: db})
+
+export default StartBlock

community-contributions/validator-report-backend/src/db/models/validatorstats.ts

@@ -0,0 +1,35 @@
+import db from '../db'
+import { DataTypes, Op, Model } from 'sequelize'
+
+class ValidatorStats extends Model {}
+
+ValidatorStats.init({
+  accountId: {type: DataTypes.INTEGER, allowNull: false},
+  eraId: {type: DataTypes.INTEGER, allowNull: false},
+  stake_total: { type: DataTypes.DECIMAL, defaultValue: 0},
+  stake_own: { type: DataTypes.DECIMAL, defaultValue: 0},
+  points: { type: DataTypes.INTEGER, defaultValue: 0},
+  rewards: { type: DataTypes.DECIMAL, defaultValue: 0},
+  commission: DataTypes.DECIMAL
+}, 
+{modelName: 'validator_stats', sequelize: db, indexes: [
+  {
+    unique: true,
+    fields: ['accountId', 'eraId']
+  }
+ ]
+})
+
+ValidatorStats.removeAttribute('id')
+
+export const findByAccountAndEra = (account: number, era: number): Promise<ValidatorStats> => {
+  return ValidatorStats.findOne( 
+    { 
+      where: {
+        accountId: { [Op.eq]: account },
+        eraId: { [Op.eq]: era }
+      }
+  })
+}
+
+export default ValidatorStats

community-contributions/validator-report-backend/src/db/native_queries.ts

@@ -0,0 +1,99 @@
+import {Moment} from 'moment'
+export const pageSize = 50
+
+export const validatorStats = (address: string, startBlock = -1, endBlock = -1, startTime: Moment, endTime: Moment, page = 1, countQuery = false): string => 
+`select 
+	${countQuery ? ' count(vs."eraId")::integer as "totalCount" ' : ` vs."eraId" as id, 
+    stake_total as "stakeTotal", 
+    stake_own as "stakeOwn", 
+    points, 
+    rewards, 
+    commission, 
+    subq2.blocks_cnt as "blocksCount" `}
+from 
+	validator_stats vs 
+inner join 
+	accounts a on a.id = vs."accountId" 
+${countQuery ? '' : `inner join 
+	(select 
+		"eraId", count(b.id) blocks_cnt 
+	from 
+		blocks b 
+	${address != '' ? ` where b."validatorId" in (select id from accounts where key = '${address}') ` : ''}
+	    group by "eraId") subq2 
+	on 
+		subq2."eraId" = vs."eraId"`} 
+where ${address != '' ? `a.key = '${address}' and ` : ''}
+	vs."eraId" in 
+	(select 
+        subq.era 
+    from 
+        (select 
+            distinct("eraId") era 
+        from blocks 
+        where ${startBlock > 0 ? ` blocks.id >= ${startBlock} and blocks.id <= ${endBlock} ` : '1 = 1'} 
+        ${startTime ? ` AND blocks.timestamp >= '${startTime.toISOString()}'::date and blocks.timestamp <= '${endTime.toISOString()}'::date ` : ''}) subq
+        ) ${countQuery ? '' : ` order by id limit ${pageSize} offset ${pageSize * (page - 1)} `}`
+
+
+export const countTotalBlocksProduced = (address: string, startBlock = -1, endBlock = -1, startTime: Moment = null, endTime: Moment = null) => 
+`SELECT count(b.id) as "totalBlocks"
+FROM blocks b 
+INNER JOIN accounts a ON a.id = b."validatorId"
+WHERE ${address != '' ? `a.key = '${address}' 
+    AND ` : ''} ${startBlock > 0 ? ` b.id >= ${startBlock} AND b.id <= ${endBlock} ` : ' 1=1 '} 
+    ${startTime ? ` AND b.timestamp >= '${startTime.toISOString()}'::date AND b.timestamp <= '${endTime.toISOString()}'::date ` : ' AND 1=1 '}`
+
+export const findBlockByTime = (timeMoment: Moment) => 
+`SELECT b.*
+FROM blocks b
+ORDER BY (ABS(EXTRACT(epoch
+    FROM (b.timestamp - '${timeMoment.toISOString()}'::date))))
+LIMIT 1`
+
+export const findFirstAuthoredBlock = (blockIdStart: number, blockIdEnd: number, addr: string) => 
+`SELECT b.*
+FROM blocks b
+INNER JOIN accounts a ON a.id = b."validatorId" AND a.key = '${addr}' 
+${blockIdStart > 0 ? ` WHERE b.id >= ${blockIdStart} AND b.id <= ${blockIdEnd} ` : ''}
+ORDER BY b.id
+LIMIT 1`
+
+export const findLastAuthoredBlock = (blockIdStart: number, blockIdEnd: number, addr: string) => 
+`SELECT b.*
+FROM blocks b
+INNER JOIN accounts a ON a.id = b."validatorId" AND a.key = '${addr}' 
+${blockIdStart > 0 ? ` WHERE b.id >= ${blockIdStart} AND b.id <= ${blockIdEnd} ` : ''}
+ORDER BY b.id DESC
+LIMIT 1`
+
+export interface IValidatorReport {
+    startBlock: number, 
+    startEra: number, 
+    endBlock: number,
+    endEra: number,
+    startTime: number,
+    endTime: number,
+    totalBlocks: number,
+    totalCount: number,
+    pageSize: number,
+    report: IValidatorEraStats[]
+}
+
+export interface IValidatorEraStats {
+    id: number,
+    stakeTotal: number,
+    stakeOwn: number,
+    points: number,
+    rewards: number,
+    commission: number,
+    blocksCount: number
+}
+
+export interface ITotalCount {
+    totalCount: number
+}
+
+export interface ITotalBlockCount {
+    totalBlocks: number
+}

community-contributions/validator-report-backend/src/db/seed.ts

@@ -0,0 +1,38 @@
+import db from './db'
+import {
+  Block,
+  Event,
+  Category,
+  Channel,
+  Council,
+  Consul,
+  ConsulStake,
+  Member,
+  Post,
+  Proposal,
+  Thread,
+} from './models'
+
+const blocks: any[] = [] //require('../../blocks.json')
+
+async function runSeed() {
+  await db.sync({ force: true })
+  console.log('db synced!')
+  console.log('seeding...')
+  try {
+    if (blocks.length) {
+      console.log('importing blocks')
+      const b = await Block.bulkCreate(blocks)
+      console.log(`${b.length} blocks imported`)
+    }
+  } catch (err) {
+    console.error(`sequelize error:`, err)
+    process.exitCode = 1
+  } finally {
+    console.log('closing db connection')
+    await db.close()
+    console.log('db connection closed')
+  }
+}
+
+runSeed()

community-contributions/validator-report-backend/src/index.ts

@@ -0,0 +1,151 @@
+import { addBlock } from './joystream'
+import { connectUpstream } from './joystream/ws'
+import express from 'express'
+import cors from 'cors'
+import ascii from './ascii'
+import db from './db'
+import moment from 'moment'
+import { QueryOptionsWithType, QueryTypes, Sequelize } from 'sequelize'
+import {
+    validatorStats, 
+    countTotalBlocksProduced, 
+    findBlockByTime, 
+    findFirstAuthoredBlock,
+    findLastAuthoredBlock,
+    IValidatorReport, 
+    ITotalCount, 
+    ITotalBlockCount, 
+    IValidatorEraStats,
+    pageSize} from './db/native_queries'
+import { Header } from './types'
+
+import {
+    Block,
+    StartBlock
+} from './db/models'
+
+const PORT: number = process.env.PORT ? +process.env.PORT : 3500
+
+const app = express()
+
+app.listen(PORT, () =>
+  console.log(`[Express] Listening on port ${PORT}`, ascii)
+)
+
+;(async () => {
+  
+    const api = await connectUpstream()
+
+    let lastHeader: Header = { number: 0, timestamp: 0, author: '' }
+    let firstProcessedBlockLogged = false
+
+
+    let highId = 0
+    Block.max('id').then(
+        (highestProcessedBlock: number) => {
+            highId = highestProcessedBlock === undefined || isNaN(highestProcessedBlock) ? 0 : highestProcessedBlock
+            StartBlock.destroy({where: {}})
+            api.rpc.chain.subscribeNewHeads(async (header: Header) => {
+                const id = +header.number
+                if (id === +lastHeader.number)
+                    return console.debug(
+                        `[Joystream] Skipping duplicate block ${id} (TODO handleFork)`
+                    )
+                lastHeader = header
+                await addBlock(api, header)
+                if(!firstProcessedBlockLogged) {
+                    StartBlock.create({block: id})
+                    console.log(`[Joystream] Subscribed to new blocks starting from ${id}`)
+                    firstProcessedBlockLogged = true
+                }
+            })
+        }
+    )
+})()
+
+app.use(cors())
+app.use(express.json())
+app.use(express.urlencoded({ extended: true }))
+
+const corsOptions = {
+    origin: process.env.ALOWED_ORIGIN || 'http://localhost:3000',
+    optionsSuccessStatus: 200
+}
+
+const ADDRESS_LENGTH = 48
+
+const opts = {type: QueryTypes.SELECT, plain: true} as QueryOptionsWithType<QueryTypes.SELECT> & { plain: true }
+
+app.get('/validator-report', cors(corsOptions), async (req: any, res: any, next: any) => {
+    try {
+        const address = (req.query.addr && req.query.addr.length == ADDRESS_LENGTH) ? req.query.addr : ''
+        const page = !isNaN(req.query.page) ? req.query.page : 1
+        const startBlock = !isNaN(req.query.start_block) ? +req.query.start_block : -1
+        const endBlock = !isNaN(req.query.end_block) ? +req.query.end_block : -1
+        console.log(`Start block = ${startBlock}, end block = ${endBlock}`)
+        if(startBlock > 0 && endBlock > 0 && endBlock > startBlock) {
+            return res.json(await fetchReportPage(
+                validatorStats(address, startBlock, endBlock, null, null, page), 
+                validatorStats(address, startBlock, endBlock, null, null, page, true), 
+                countTotalBlocksProduced(address, startBlock, endBlock),
+                findFirstAuthoredBlock(startBlock, endBlock, address),
+                findLastAuthoredBlock(startBlock, endBlock, address)
+            ))
+        } else {
+
+            const startTime = moment.utc(req.query.start_time, 'YYYY-MM-DD').startOf('d')
+            const endTime = moment.utc(req.query.end_time, 'YYYY-MM-DD').endOf('d')
+            console.log(`Start time: [${startTime}]-[${endTime}]`)
+            if(endTime.isAfter(startTime)) {
+                return res.json(await fetchReportPage(
+                    validatorStats(address, -1, -1, startTime, endTime, page), 
+                    validatorStats(address, -1, -1, startTime, endTime, page, true), 
+                    countTotalBlocksProduced(address, -1, -1, startTime, endTime),
+                    findBlockByTime(startTime), 
+                    findBlockByTime(endTime) 
+                  ))
+            } else {
+                return res.json(await fetchReportPage(
+                    validatorStats(address, -1, -1, null, null, page), 
+                    validatorStats(address, -1, -1, null, null, page, true), 
+                    countTotalBlocksProduced(address),
+                    findFirstAuthoredBlock(startBlock, endBlock, address),
+                    findLastAuthoredBlock(startBlock, endBlock, address)
+                    ))
+              }
+            }
+    } catch (err) {
+        console.log(err)
+        return res.json({})
+    }
+  })
+
+const fetchReportPage = async (
+    validatorStatsSql: string, 
+    validatorStatsCountSql: string, 
+    totalBlocksSql: string,
+    firstBlockSql: string,
+    lastBlockSql: string,
+    ): Promise<IValidatorReport> => {
+
+    const dbBlockStart = (await db.query<any>(firstBlockSql, opts)) // TODO <Block> instead of <any> produces an object with no get() function  
+    const dbBlockEnd = (await db.query<any>(lastBlockSql, opts))
+    const dbCount = (await db.query<ITotalCount>(validatorStatsCountSql, opts))
+    const blockCount = (await db.query<ITotalBlockCount>(totalBlocksSql, opts))
+
+    return db.query<IValidatorEraStats>(validatorStatsSql, {type: QueryTypes.SELECT}).then((stats: IValidatorEraStats[]) => {
+        const validationReport: IValidatorReport = {
+            pageSize: pageSize,
+            totalCount: dbCount.totalCount,
+            startBlock: dbBlockStart?.id,
+            endBlock: dbBlockEnd?.id,
+            startTime: dbBlockStart?.timestamp,
+            endTime: dbBlockEnd?.timestamp,
+            startEra: dbBlockStart?.eraId,
+            endEra: dbBlockEnd?.eraId,
+            totalBlocks: blockCount.totalBlocks | 0,
+            report: stats
+        }
+        return validationReport
+    })
+}

community-contributions/validator-report-backend/src/init_db.ts

@@ -0,0 +1,11 @@
+import db from './db'
+
+async function main () {
+
+    await db.sync({ force: true })
+    console.log('DB created')
+
+}
+
+main()
+

community-contributions/validator-report-backend/src/joystream/index.ts

@@ -0,0 +1,268 @@
+import {
+  Account,
+  Block,
+  Era,
+  Event,
+  ValidatorStats
+} from '../db/models'
+
+import moment from 'moment'
+import chalk from 'chalk'
+
+import { HeaderExtended } from '@polkadot/api-derive/type/types';
+import {
+  Api,
+} from '../types'
+
+
+import {
+  AccountId,
+  Moment,
+  EventRecord,
+  BlockHash,
+} from '@polkadot/types/interfaces'
+import { Vec } from '@polkadot/types'
+import { ApiPromise } from '@polkadot/api';
+
+let queue: any[] = []
+let processing = ''
+
+export const processNext = async () => {
+  const task = queue.shift()
+  if (!task) return
+  await task()
+  processNext()
+}
+
+const accounts = new Map<string, any>()
+
+const getBlockHash = (api: ApiPromise, blockId: number) =>
+  api.rpc.chain.getBlockHash(blockId).then((hash: BlockHash) => hash.toString())
+
+const getEraAtHash = (api: ApiPromise, hash: string) =>
+  api.query.staking.activeEra
+    .at(hash)
+    .then((era) => era.unwrap().index.toNumber())
+
+const getAccount = async (address: string) => {
+    if (accounts.get(address)) {
+      return accounts.get(address)
+    } else {
+      const account = (await Account.findOrCreate({where: {key : address}}))[0].get({plain: true})
+      accounts.set(address, account)
+      return account
+    }
+}
+
+const getEraAtBlock = async (api: ApiPromise, block: number) =>
+  getEraAtHash(api, await getBlockHash(api, block))
+
+const getTimestamp = async (api: ApiPromise, hash?: string) => {
+  const timestamp = hash
+    ? await api.query.timestamp.now.at(hash)
+    : await api.query.timestamp.now()
+  return moment.utc(timestamp.toNumber()).valueOf()
+}
+
+export const addBlock = async (
+  api: ApiPromise,
+  header: { number: number; author: string }
+) => {
+  const id = +header.number
+  const exists = await Block.findByPk(id) 
+  if (exists) {
+    console.error(`TODO handle fork`, String(header.author))
+  }
+
+  await processBlock(api, id)
+  
+  // logging
+  //const handle = await getHandleOrKey(api, key)
+  const q = queue.length ? chalk.green(` [${queue.length}:${processing}]`) : ''
+  console.log(`[Joystream] block ${id} ${q}`)
+}
+
+const processBlock = async (api: ApiPromise, id: number) => {
+
+  const exists = (await Block.findByPk(id))
+  if (exists) return exists.get({plain: true})
+
+  processing = `block ${id}`
+  console.log(processing)
+
+  const previousBlockModel = (await Block.findByPk(id - 1))
+  let lastBlockTimestamp = 0
+  let lastBlockHash = ''
+  let lastEraId = 0
+  if (previousBlockModel) {
+    const previousDbBlock = previousBlockModel.get({plain: true})
+    lastBlockTimestamp = previousDbBlock.timestamp.getTime();
+    lastBlockHash = previousDbBlock.hash
+    lastEraId = previousDbBlock.eraId
+  } else {
+    lastBlockHash = await getBlockHash(api, id - 1);
+    lastBlockTimestamp = await getTimestamp(api, lastBlockHash);
+    lastEraId = await getEraAtHash(api, lastBlockHash)
+  }
+  const hash = await getBlockHash(api, id)
+  const currentBlockTimestamp = await getTimestamp(api, hash)
+  const extendedHeader = await api.derive.chain.getHeader(hash) as HeaderExtended
+
+  const eraId = await getEraAtHash(api, hash)
+  let chainTime
+  if(eraId - lastEraId === 1) {
+    console.log('This block marks the start of new era. Updating the previous era stats')
+    const {total, individual} = await api.query.staking.erasRewardPoints.at(lastBlockHash, lastEraId)
+    const slots = (await api.query.staking.validatorCount.at(lastBlockHash)).toNumber()
+    const newEraTime = (await api.query.timestamp.now.at(hash)) as Moment
+    chainTime = moment(newEraTime.toNumber())
+
+    await Era.upsert({ // update stats for previous era
+      id: lastEraId,
+      slots: slots,
+      stake: await api.query.staking.erasTotalStake.at(hash, lastEraId),
+      eraPoints: total
+    })
+
+
+    const validatorStats = await ValidatorStats.findAll({where: {eraId: lastEraId}, include: [Account]})
+    for (let stat of validatorStats) {
+      const validatorStats = stat.get({plain: true})
+      const validatorAccount = validatorStats.account.key
+      console.log(validatorAccount)
+      let pointVal = 0;
+      for(const [key, value] of individual.entries()) {
+        if(key == validatorAccount) {
+          pointVal = value.toNumber()
+          break
+        }
+      }
+
+      const {total, own} = await api.query.staking.erasStakers.at(lastBlockHash, lastEraId, validatorAccount)
+
+      ValidatorStats.upsert({
+        eraId: lastEraId, 
+        accountId: validatorStats.accountId, 
+        stake_own: own, 
+        stake_total: total, 
+        points: pointVal,
+        commission: (await api.query.staking.erasValidatorPrefs.at(lastBlockHash, eraId, validatorAccount)).commission.toNumber() / 10000000
+      })
+    }
+  
+  }
+  const [era, created] = await Era.upsert({ // add the new are with just a timestamp of its first block
+    id: eraId,
+    timestamp: chainTime
+  }, {returning: true})
+
+  const block = await Block.upsert({
+    id: id, 
+    hash: hash,
+    timestamp: moment.utc(currentBlockTimestamp).toDate(),
+    blocktime: (currentBlockTimestamp - lastBlockTimestamp),
+    eraId: era.get({plain: true}).id,
+    validatorId: (await getAccount(extendedHeader.author.toHuman())).id
+  }, {returning: true})
+
+  await importEraAtBlock(api, id, hash, era)
+  processEvents(api, id, eraId, hash)
+
+  return block
+}
+
+export const addBlockRange = async (
+  api: ApiPromise,
+  startBlock: number,
+  endBlock: number
+) => {
+  for (let block = startBlock; block <= endBlock; block++) {
+    queue.push(() => processBlock(api, block))
+  }
+}
+
+const processEvents = async (api: ApiPromise, blockId: number, eraId: number, hash: string) => {
+  processing = `events block ${blockId}`
+  try {
+    const blockEvents = await api.query.system.events.at(hash)
+    blockEvents.forEach(async ({ event }: EventRecord) => {
+      let { section, method, data } = event
+      if(section == 'staking' && method == 'Reward') {
+        const addressCredited = data[0].toString()
+        await Event.create({ blockId, section, method, data: JSON.stringify(data) })
+        Account.findOne(
+          {
+            where: {
+              key: addressCredited
+            }
+          }
+        ).then(async (beneficiaryAccount: Account) => {
+          let address = ''
+          if (beneficiaryAccount == null) {
+            address = (await Account.create({key: addressCredited}, {returning: true})).get({plain: true}).id
+          } else {
+            address = beneficiaryAccount.get({plain: true}).id
+          }
+          await ValidatorStats.upsert(
+            {
+              accountId: address, 
+              eraId: eraId,
+              rewards: Number(data[1])
+            }
+          )  
+        })
+      }
+    })
+  } catch (e) {
+    console.log(`failed to fetch events for block  ${blockId} ${hash}`)
+  }
+}
+
+
+const importEraAtBlock = async (api: Api, blockId: number, hash: string, eraModel: Era) => {
+  const era = eraModel.get({plain: true})
+  if (era.active) return
+  const id = era.id
+  processing = `era ${id}`
+  try {
+    const snapshotValidators = await api.query.staking.snapshotValidators.at(hash);
+    if (!snapshotValidators.isEmpty) {
+      console.log(`[Joystream] Found validator info for era ${id}`)
+
+      const validators = snapshotValidators.unwrap() as Vec<AccountId>;
+      const validatorCount = validators.length
+  
+      for (let validator of validators) {
+        // create stub records, which will be populated with stats on the first block of the next era
+        ValidatorStats.upsert({
+          eraId: id, 
+          accountId: (await getAccount(validator.toHuman())).id, 
+        })
+      }
+  
+      const slots = (await api.query.staking.validatorCount.at(hash)).toNumber()
+  
+      await Era.upsert({
+        id: id,
+        allValidators: validatorCount,
+        waitingValidators: validatorCount > slots ? validatorCount - slots : 0,
+      })  
+    }
+
+    const snapshotNominators = await api.query.staking.snapshotNominators.at(hash);
+    if (!snapshotNominators.isEmpty) {
+      const nominators = snapshotNominators.unwrap() as Vec<AccountId>;
+      await Era.upsert({
+        id: id,
+        nominators: nominators.length
+      })  
+    }
+    return id;
+
+  } catch (e) {
+    console.error(`import era ${blockId} ${hash}`, e)
+  }
+}
+
+
+module.exports = { addBlock, addBlockRange, processNext }

community-contributions/validator-report-backend/src/joystream/ws.ts

@@ -0,0 +1,24 @@
+// TODO allow alternative backends
+
+import { ApiPromise, WsProvider } from '@polkadot/api'
+import { types } from '@joystream/types'
+
+const wsLocation =
+    process.env.RPC_ENDPOINT || 'ws://localhost:9944'
+    // 'wss://rome-rpc-endpoint.joystream.org:9944'
+
+export const connectUpstream = async (): Promise<ApiPromise> => {
+    try {
+        //console.debug(`[Joystream] Connecting to ${wsLocation}`)
+        const provider = new WsProvider(wsLocation)
+        const api = await ApiPromise.create({ provider, types })
+        await api.isReady
+        console.debug(`[Joystream] Connected to ${wsLocation}`)
+        return api
+    } catch (e) {
+        console.error(`[Joystream] upstream connection failed`, e)
+        throw new Error()
+    }
+}
+
+module.exports = { connectUpstream }

community-contributions/validator-report-backend/src/socket.ts

@@ -0,0 +1,7 @@
+import io from "socket.io-client";
+
+const socket_location = `${process.env.PUBLIC_URL}/socket.io`;
+const socket = io(window.location.origin, { path: socket_location });
+console.log(`socket location: ${window.location.origin + socket_location}`);
+
+export default socket;

community-contributions/validator-report-backend/src/types.js

@@ -0,0 +1,217 @@
+"use strict";
+exports.__esModule = true;
+exports.CacheEvent = exports.Bounty = exports.Channel = exports.Media = exports.MintStatistics = exports.SpendingProposals = exports.ProposalTypes = exports.Exchange = exports.WorkersInfo = exports.ValidatorReward = exports.Statistics = void 0;
+var Statistics = /** @class */ (function () {
+    function Statistics() {
+        this.councilRound = 0;
+        this.councilMembers = 0;
+        this.electionApplicants = 0;
+        this.electionAvgApplicants = 0;
+        this.perElectionApplicants = 0;
+        this.electionApplicantsStakes = 0;
+        this.electionVotes = 0;
+        this.avgVotePerApplicant = 0;
+        this.dateStart = "";
+        this.dateEnd = "";
+        this.startBlock = 0;
+        this.endBlock = 0;
+        this.percNewBlocks = 0;
+        this.startMembers = 0;
+        this.endMembers = 0;
+        this.newMembers = 0;
+        this.percNewMembers = 0;
+        this.newBlocks = 0;
+        this.avgBlockProduction = 0;
+        this.startThreads = 0;
+        this.endThreads = 0;
+        this.newThreads = 0;
+        this.totalThreads = 0;
+        this.percNewThreads = 0;
+        this.startPosts = 0;
+        // endPosts: number = 0;
+        this.newPosts = 0;
+        this.endPosts = 0;
+        this.percNewPosts = 0;
+        this.startCategories = 0;
+        this.endCategories = 0;
+        this.newCategories = 0;
+        this.perNewCategories = 0;
+        this.newProposals = 0;
+        this.newApprovedProposals = 0;
+        this.startChannels = 0;
+        this.newChannels = 0;
+        this.endChannels = 0;
+        this.percNewChannels = 0;
+        this.startMedia = 0;
+        this.newMedia = 0;
+        this.endMedia = 0;
+        this.percNewMedia = 0;
+        this.deletedMedia = 0;
+        this.newMints = 0;
+        this.startMinted = 0;
+        this.totalMinted = 0;
+        this.percMinted = 0;
+        this.endMinted = 0;
+        this.totalMintCapacityIncrease = 0;
+        this.startCouncilMinted = 0;
+        this.endCouncilMinted = 0;
+        this.newCouncilMinted = 0;
+        this.percNewCouncilMinted = 0;
+        this.startCuratorMinted = 0;
+        this.endCuratorMinted = 0;
+        this.newCuratorMinted = 0;
+        this.percCuratorMinted = 0;
+        this.startStorageMinted = 0;
+        this.endStorageMinted = 0;
+        this.newStorageMinted = 0;
+        this.percStorageMinted = 0;
+        this.startIssuance = 0;
+        this.endIssuance = 0;
+        this.newIssuance = 0;
+        this.percNewIssuance = 0;
+        this.newTokensBurn = 0;
+        this.newValidatorRewards = 0;
+        this.avgValidators = 0;
+        this.startValidators = "";
+        this.endValidators = "";
+        this.percValidators = 0;
+        this.startValidatorsStake = 0;
+        this.endValidatorsStake = 0;
+        this.percNewValidatorsStake = 0;
+        this.startStorageProviders = 0;
+        this.endStorageProviders = 0;
+        this.percNewStorageProviders = 0;
+        this.newStorageProviderReward = 0;
+        this.startStorageProvidersStake = 0;
+        this.endStorageProvidersStake = 0;
+        this.percNewStorageProviderStake = 0;
+        this.newCouncilRewards = 0;
+        this.startCurators = 0;
+        this.endCurators = 0;
+        this.percNewCurators = 0;
+        this.newCuratorRewards = 0;
+        this.startUsedSpace = 0;
+        this.newUsedSpace = 0;
+        this.endUsedSpace = 0;
+        this.percNewUsedSpace = 0;
+        this.avgNewSizePerContent = 0;
+        this.totalAvgSizePerContent = 0;
+        this.percAvgSizePerContent = 0;
+        this.newStakes = 0;
+        this.totalNewStakeValue = 0;
+        this.newTextProposals = 0;
+        this.newRuntimeUpgradeProposal = 0;
+        this.newSetElectionParametersProposal = 0;
+        this.spendingProposalsTotal = 0;
+        this.bountiesTotalPaid = 0;
+        this.newSetLeadProposal = 0;
+        this.newSetContentWorkingGroupMintCapacityProposal = 0;
+        this.newEvictStorageProviderProposal = 0;
+        this.newSetValidatorCountProposal = 0;
+        this.newSetStorageRoleParametersProposal = 0;
+    }
+    return Statistics;
+}());
+exports.Statistics = Statistics;
+var ValidatorReward = /** @class */ (function () {
+    function ValidatorReward() {
+        this.sharedReward = 0;
+        this.remainingReward = 0;
+        this.validators = 0;
+        this.slotStake = 0;
+        this.blockNumber = 0;
+    }
+    return ValidatorReward;
+}());
+exports.ValidatorReward = ValidatorReward;
+var WorkersInfo = /** @class */ (function () {
+    function WorkersInfo() {
+        this.rewards = 0;
+        this.startStake = 0;
+        this.endStake = 0;
+        this.startNrOfWorkers = 0;
+        this.endNrOfWorkers = 0;
+    }
+    return WorkersInfo;
+}());
+exports.WorkersInfo = WorkersInfo;
+var Exchange = /** @class */ (function () {
+    function Exchange() {
+        this.sender = "";
+        this.amount = 0;
+        this.fees = 0;
+        this.blockNumber = 0;
+    }
+    return Exchange;
+}());
+exports.Exchange = Exchange;
+var ProposalTypes;
+(function (ProposalTypes) {
+    ProposalTypes["Text"] = "Text";
+    ProposalTypes["RuntimeUpgrade"] = "RuntimeUpgrade";
+    ProposalTypes["SetElectionParameters"] = "SetElectionParameters";
+    ProposalTypes["Spending"] = "Spending";
+    ProposalTypes["SetLead"] = "SetLead";
+    ProposalTypes["SetContentWorkingGroupMintCapacity"] = "SetContentWorkingGroupMintCapacity";
+    ProposalTypes["EvictStorageProvider"] = "EvictStorageProvider";
+    ProposalTypes["SetValidatorCount"] = "SetValidatorCount";
+    ProposalTypes["SetStorageRoleParameters"] = "SetStorageRoleParameters";
+})(ProposalTypes = exports.ProposalTypes || (exports.ProposalTypes = {}));
+var SpendingProposals = /** @class */ (function () {
+    function SpendingProposals(id, spentAmount) {
+        this.id = id;
+        this.spentAmount = spentAmount;
+    }
+    return SpendingProposals;
+}());
+exports.SpendingProposals = SpendingProposals;
+var MintStatistics = /** @class */ (function () {
+    function MintStatistics(startMinted, endMinted, diffMinted, percMinted) {
+        if (startMinted === void 0) { startMinted = 0; }
+        if (endMinted === void 0) { endMinted = 0; }
+        if (diffMinted === void 0) { diffMinted = 0; }
+        if (percMinted === void 0) { percMinted = 0; }
+        this.startMinted = startMinted;
+        this.endMinted = endMinted;
+        this.diffMinted = diffMinted;
+        this.percMinted = percMinted;
+    }
+    return MintStatistics;
+}());
+exports.MintStatistics = MintStatistics;
+var Media = /** @class */ (function () {
+    function Media(id, title) {
+        this.id = id;
+        this.title = title;
+    }
+    return Media;
+}());
+exports.Media = Media;
+var Channel = /** @class */ (function () {
+    function Channel(id, title) {
+        this.id = id;
+        this.title = title;
+    }
+    return Channel;
+}());
+exports.Channel = Channel;
+var Bounty = /** @class */ (function () {
+    function Bounty(proposalId, title, status, amountAsked, amountMinted) {
+        this.proposalId = proposalId;
+        this.title = title;
+        this.status = status;
+        this.amountAsked = amountAsked;
+        this.amountMinted = amountMinted;
+    }
+    return Bounty;
+}());
+exports.Bounty = Bounty;
+var CacheEvent = /** @class */ (function () {
+    function CacheEvent(section, method, data) {
+        this.section = section;
+        this.method = method;
+        this.data = data;
+    }
+    return CacheEvent;
+}());
+exports.CacheEvent = CacheEvent;

community-contributions/validator-report-backend/tsconfig.json

@@ -0,0 +1,39 @@
+{
+  "compilerOptions": {
+    "target": "esnext",
+    "module": "commonjs",
+    "strict": false,
+    "sourceMap": true,
+    "noImplicitAny": false,
+    "noUnusedLocals": false,
+    "noImplicitReturns": true,
+    "moduleResolution": "node",
+    "allowSyntheticDefaultImports": true,     /* Allow default imports from modules with no default export. This does not affect code emit, just typechecking. */
+    "esModuleInterop": true,                  /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */
+    "experimentalDecorators": true,           /* Enables experimental support for ES7 decorators. */
+    "declaration": true,
+    "resolveJsonModule": true,
+    "types" : [
+      "node"
+    ],
+    "forceConsistentCasingInFileNames": true,
+    "baseUrl": ".",
+    "paths": {
+      "@polkadot/types/augment": ["./node_modules/@joystream/types/augment-codec/augment-types.ts"]
+    },
+    "typeRoots": [
+      "./node_modules/@polkadot/ts",
+      "./node_modules/@types"
+    ],
+    "declarationDir": "lib",
+    "outDir": "lib"
+  },
+  "include": [
+    "src/*.ts"
+  ],
+  "exclude": [
+    "node_modules",
+    "**/*.spec.ts",
+    "**/*.d.ts"
+  ]
+}