nebula crawler chart

[coryschwartz]

helm chart to run the nebula crawler

[dennis-tra]

If you changed the env name on the other repo this var needed to be NEBULA_DATABASE_SSL

[dennis-tra]

If you also want to collect neighbor information you may consider to add the --neighbors flag. This will, however, roughly double the data it collects on each crawl.

[dennis-tra]

I may suggest to incorporate the network name that nebula is crawling/monitoring into the naming schemes. AFAICT This Helm chart only considers IPFS. If you also want to crawl/monitor Filecoin in the future you may include the network name in the Kubernetes resource names as well.

Like

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  # name: {{ .Release.Name }}-crawl
  name: {{ .Release.Name }}-ipfs-crawl

[dennis-tra]

Just leaving a few numbers here:

IPFS (from 26th of November 2021 until today):

• db size: 36 GB
• crawls: 1962
• -> ~18 MB/crawl
• just the neighbors table: 18 GB
• If you exclude the neighbors collection -> ~9MB/crawl.
  The IPFS network doubled in size during December so the MB/crawl may actually be higher starting from now on as the above numbers are averaged over the period of December.

Filecoin (from 26th of November 2021 until today):

• db size: 6586 MB
• crawls: 1963
• -> 3.4 MB/crawl
• just the neighbors table: 3761 MB
• If you exclude the neighbors collection -> 1.4 MB/crawl