l1.media ba7dbd7331 Archive testnet SWG, DWG reports (#867) 1 год назад
..
README.md ba7dbd7331 Archive testnet SWG, DWG reports (#867) 1 год назад
ids.js ba7dbd7331 Archive testnet SWG, DWG reports (#867) 1 год назад
package.json ba7dbd7331 Archive testnet SWG, DWG reports (#867) 1 год назад
verify.js ba7dbd7331 Archive testnet SWG, DWG reports (#867) 1 год назад

README.md

41.CC-1 - Finalizing Content Migration

The goal is to check if transferred channels and videos from Sumer testnet are available on Giza distributors.

Abstract

After the upgrade, we migrated a lot of the content from sumer over to giza. This means some follow up work is required. Based on our own, and the community input, the channels that was migrated (including all videos) is available in thi issue. For addressing both this, and the KPI 41.CC-2, using the "old" and the "new" query-node is needed.

Find out if all of the following was in fact migrated as we inteded, or if something was missing: channels videos (he migrations says yes for both, but double check wouldn't hurt) assets - some nuances here: How many ACCEPTED (ignore PENDING/REJECTED) data objects were there in total stored under these channels? How many bytes in total? We know the exact amounts migrated. For 1. Share the queries and scripts used to get the data.

verify.js does following

  • load ids.js with channel, video mapping
  • query and cache Sumer QN for channels (oldChannels.json)
  • query and cache Giza QN for videos (newVideos.json)
  • select migrated ids and perform one HEAD request for each asset on https://storage-1.joystream.org/argus (~12 results/s, no throttling, sorry!)
  • save results to migration_result.json
  • print tables for empty source videos (PENDING) and non-uploads on Giza (see below).

Were there any censored channels or videos that made it into the migration? If yes, re-evaluate, and censor again if needed.

4 channels (54 55 57 133) are marked as isCensored, see Censored source channels.

Are there any channels or videos that wasn't attempted migrated, but should have been? Make a channel on discord, and a forum thread where users can post what they are missing. Review what is being requested, and share the results.

Of 4706 selected videos 4646 were successfully transferred (646,5 GB video data + 82 MB thumbnails).

60 empty source videos were selected for transfer and the same amount is empty on the target provider.

Resources

Queries

Channel <> Videos collection Sumer

query {
  channels (where:{id_eq:534 }) {
    id
    createdAt
    updatedAt
    createdInBlock
    title
    category {
      name
    }
    ownerMember {
      id
      handle
      channels {
        id
        title
      }
    }
    coverPhotoAvailability
    avatarPhotoAvailability
    coverPhotoDataObject {
      joystreamContentId
      size
    }
    avatarPhotoDataObject {
      joystreamContentId
      size
    }
    isCensored
    isPublic
    }
    languages {
      iso
    }
    videos {
      id
      title
      mediaAvailability
      thumbnailPhotoAvailability
    }
}

Shorter version here

Assets on JSG provider (Giza)

query{storageBuckets(where:{id_eq:"1"}){id,createdAt,acceptingNewBags,dataObjectsSizeLimit,dataObjectCountLimit,dataObjectsCount,dataObjectsSize,bags{id,objects{id,createdAt,isAccepted,size}}operatorMetadata{id,nodeEndpoint,nodeLocationId,extra,nodeLocation{id,countryCode,coordinates{latitude,longitude}}}}}

=> https://discord.com/channels/811216481340751934/812344681786507274/936573775066054688

Channel <> Videos collection and Bag <> StorageBucket associations Giza

query {
  storageBags { id distributionBuckets{id familyId bucketIndex} }  
  channels {id createdAt updatedAt createdInBlock title
    category { name } isCensored isPublic
    ownerMember { id handle }
    videos { id title }
  }
}

Results

Loaded IDs of 4706 transferred videos in 177 channels. Loaded info of 4706 migrated videos. Loaded info of 713 old channels. 4646 videos were transferred 646485444380 bytes + 82260654 avatars + covers.

Migrated videos with empty source

ID Title Size Upload Thumbnail
6154 test upload 140534084 PENDING ACCEPTED
296 asd 253791949 PENDING ACCEPTED
298 2021-05-25 11-13-38 253791949 PENDING ACCEPTED
299 2021-05-25 11-13-38 253791949 PENDING ACCEPTED
300 2021-05-25 11-13-38 253791949 PENDING ACCEPTED
301 asdasd 253791949 PENDING ACCEPTED
2040 Как стать Амбассадором Bit.Country 144893665 PENDING ACCEPTED
120 Showreel animation 2021 44997716 PENDING PENDING
121 Showreel animation 2021 44997716 PENDING PENDING
303 Tretyakov Gallery - Black Square 755357428 PENDING PENDING
206 Любимое кафе с лучшим видом 19237360 PENDING PENDING
219 Путешествую с Пятницей 14891181 PENDING PENDING
393 Ночую в заброшке 27508553 PENDING PENDING
407 Нашел работу в горах 37731273 PENDING PENDING
413 Где я жил в кемпинге 48027398 PENDING PENDING
415 Лес обезьян 34139670 PENDING PENDING
428 Подношения 42596459 PENDING PENDING
220 Путешествую с Пятницей 14891181 PENDING PENDING
201 Как живет семья Бали 29098159 PENDING PENDING
194 Бали открыли 34331478 PENDING PENDING
216 Прекрасный вид на 360 36176482 PENDING PENDING
388 Танцы с Пятницей 36599530 PENDING PENDING
430 С Пятницей на вулкан 36065310 PENDING PENDING
399 Бали в стиле старых фильмов 79158922 PENDING PENDING
2601 JOY 6264500 PENDING ACCEPTED
1255 Superman: Eleventh Hour 34823272 PENDING ACCEPTED
1270 Superman: Jungle Drums 49888116 PENDING ACCEPTED
2388 The Great Piggy Bank Robbery 80195964 PENDING ACCEPTED
5591 I DID A INTENSE WORKOUT CHALLENGE 299462853 PENDING ACCEPTED
5593 Super-Pump Arm Workout for Mass 263483999 PENDING ACCEPTED
2473 Cosmos Laundromat 76159780 PENDING ACCEPTED
2488 Elephants Dream 69682629 PENDING ACCEPTED
2374 Big Buck Bunny 94054804 PENDING ACCEPTED
2377 Caminandes1 4999593 PENDING ACCEPTED
3511 Самые смешные коты 45902976 PENDING ACCEPTED
3513 Приколы с Котами Топовая Подборка 188260671 PENDING ACCEPTED
3522 Самые смешные коты рунета 36525391 PENDING ACCEPTED
2532 Что происходит на дорогах. 74585760 PENDING ACCEPTED
3252 Секрет обычной ТЁРКИ! 99776942 PENDING ACCEPTED
3260 Без помощи сварки 166481690 PENDING ACCEPTED
2731 Добрые поступки на дороге !! 95672013 PENDING ACCEPTED
2784 Сначало кушал один ...... 60910695 PENDING ACCEPTED
2980 ОШИБКИ при раскладке ПЛИТКИ! 54067149 PENDING ACCEPTED
2983 Резка кафельной плитки без плиткореза 25852047 PENDING ACCEPTED
3680 Убойные приколы с животными#6 69258194 PENDING ACCEPTED
3791 Милые животные 42696740 PENDING ACCEPTED
3790 Милые белки 303573777 PENDING ACCEPTED
3789 Спасение котенка 57380709 PENDING ACCEPTED
4957 ЩЕНОК ЮМИ ЧУ 149896221 PENDING ACCEPTED
4956 ТАЙНАЯ ЖИЗНЬ ДОМАШНИХ ЖИВОТНЫХ 27674316 PENDING ACCEPTED
4963 Супер коты 46215175 PENDING ACCEPTED
4997 Смешные кошки 41456550 PENDING ACCEPTED
5070 СМЕШНЫЕ ЖИВОТНЫЕ , КОТЫ СОБАКИ И ДРУГИЕ 121691971 PENDING ACCEPTED
8662 IMG_ 42613049 PENDING ACCEPTED
4684 J. Grunsfeld about working on Hubble 42109276 PENDING PENDING
5259 Funny Parrot Talking Videos 28707385 PENDING ACCEPTED
5755 Шоу 15 этажей 259660035 PENDING ACCEPTED
7724 Joystream 154651016 PENDING ACCEPTED
4854 Как собирать семантику (запросы) 216125363 PENDING ACCEPTED
8968 roksolana joy 14286490 PENDING ACCEPTED

Censored source channels

Found 4 censored channels in migration set:

{"id":"55","createdAt":"2021-05-28T15:41:12.000Z","updatedAt":"2021-06-03T07:31:00.000Z","createdInBlock":733904,"title":"Gingah","category":null,"ownerMember":{"id":"2195","handle":"gingah"},"isCensored":true,"isPublic":true,"coverPhotoAvailability":"ACCEPTED","coverPhotoDataObject":{"joystreamContentId":"5GpiUscQLtvBzZE1Gk53wvw8gz8zjP7TqFvM5bGhYZUcwCbd","size":193858},"avatarPhotoAvailability":"ACCEPTED","avatarPhotoDataObject":{"joystreamContentId":"5DyCJaqLMDxC1TgRjuGoFL3VDcw3dV9P1b3NsrdNapqyDPC7","size":15103},"videos":[]}
{"id":"57","createdAt":"2021-05-28T15:44:06.000Z","updatedAt":"2021-06-03T07:31:42.000Z","createdInBlock":733933,"title":"Gingah","category":null,"ownerMember":{"id":"2195","handle":"gingah"},"isCensored":true,"isPublic":true,"coverPhotoAvailability":"ACCEPTED","coverPhotoDataObject":{"joystreamContentId":"5HW2XWFDEgxmmGN7FQMeX52HHM8EyhbbHSp9YvGm6VCCzAxF","size":172320},"avatarPhotoAvailability":"ACCEPTED","avatarPhotoDataObject":{"joystreamContentId":"5Fg31k7kmX6Ms5oy75P3KHQ844CfrTo3FvfjeKGQHeb4gi3H","size":14980},"videos":[]}
{"id":"54","createdAt":"2021-05-28T15:40:54.000Z","updatedAt":"2021-06-03T07:24:18.000Z","createdInBlock":733901,"title":"Gingah","category":null,"ownerMember":{"id":"2195","handle":"gingah"},"isCensored":true,"isPublic":true,"coverPhotoAvailability":"ACCEPTED","coverPhotoDataObject":{"joystreamContentId":"5Cpg2fhvHtMNcrHrJ3VZ78wo1gCM47M6cKxZwGTiK23jWxzH","size":193858},"avatarPhotoAvailability":"ACCEPTED","avatarPhotoDataObject":{"joystreamContentId":"5FFTicA5vjzgM68QCNzFe8xGXxjVBq1phCRigGdXMMWCGWMU","size":15103},"videos":[]}
{"id":"133","createdAt":"2021-06-08T16:07:12.000Z","updatedAt":"2021-06-20T12:08:00.014Z","createdInBlock":892380,"title":"testing channel","category":null,"ownerMember":{"id":"2269","handle":"member_test_2112"},"isCensored":true,"isPublic":true,"coverPhotoAvailability":"INVALID","coverPhotoDataObject":null,"avatarPhotoAvailability":"INVALID","avatarPhotoDataObject":null,"videos":[]}

Verification

Performed HEAD request on 4697 videos. 4646 of 4706 assets are available.

60 have no content.

Channel Video Asset Status
734 9724 no media
734 9731 no media
734 9750 no media
734 9746 no media
734 9749 no media
734 9843 no media
734 9840 no media
734 9827 no media
734 9857 no media
734 9859 no media
734 9736 no media
734 9845 no media
734 9822 no media
734 9832 no media
747 12100 no media
751 9698 no media
751 9777 no media
751 9697 no media
757 9772 no media
757 9773 no media
757 9774 no media
757 9775 no media
757 9776 no media
758 10667 no media
777 10917 no media
778 10154 no media
778 10169 no media
784 10831 no media
799 10821 no media
799 10822 no media
799 10862 no media
799 10866 no media
801 11340 no media
801 11341 no media
801 11342 no media
801 10888 no media
801 10993 no media
801 11109 no media
801 11111 no media
801 11023 no media
801 11164 no media
801 11603 no media
801 11604 no media
801 11606 no media
801 11170 no media
801 11614 no media
801 11231 no media
801 11222 no media
801 11224 no media
801 11631 no media
801 11308 no media
814 11506 no media
819 11568 no media
821 11702 no media
822 11824 no media
822 11826 no media
828 11884 no media
864 13757 no media
869 13024 no media
889 14004 no media