对存储系统进行初步测试 龙芯3C5000

 lscpu 输出

Architecture:          loongarch64
  CPU op-mode(s):      32-bit, 64-bit
  Address sizes:       48 bits physical, 48 bits virtual
  Byte Order:          Little Endian
CPU(s):                32
  On-line CPU(s) list: 0-31
BIOS Vendor ID:        Loongson
Model name:            Loongson-3C5000
  BIOS Model name:     Loongson-3C5000 Not Specified CPU @ 2.2GHz
  BIOS CPU family:     607
  CPU family:          Loongson-64bit
  Model:               0x11
  Thread(s) per core:  1
  Core(s) per socket:  16
  Socket(s):           2
  BogoMIPS:            4400,00
  Flags:               cpucfg lam ual fpu lsx lasx crc32 lvz lbt_x86 lbt_arm lbt_mips
Caches (sum of all):   
  L1d:                 2 MiB (32 instances)
  L1i:                 2 MiB (32 instances)
  L2:                  8 MiB (32 instances)
  L3:                  64 MiB (2 instances)
NUMA:                  
  NUMA node(s):        2
  NUMA node0 CPU(s):   0-15
  NUMA node1 CPU(s):   16-31

 

磁  盘 INTEL SSDSC2KG240G8 

文件系统 zfs 版本 2.2.6-1

128k 块大小zfs 数据集 

负载曲线 fio random write 4k 块大小 24 个线程

fio --filename=/rpool/test_128k/test_file_fio_1 --rw=randwrite --bs=4k --size=10G --direct=1 --ioengine=libaio --iodepth=32 --time_based --group_reporting --name=rand-write-4k --numjobs=24 --runtime=300

写入测试结果: IOPS=133k

限制因素是 CPU + OS 性能

所有芯体均均匀受力 

How to TCPdump in Kubernetes via ephemeral container

 kubectl debug -it ingress-nginx-controller --image=dockersec/tcpdump --target controller -- sh

 

https://cloudyuga.guru/blogs/how-to-tcpdump-in-kubernetes/

Build Airflow docker image with R

dockerfile:

FROM apache/airflow:2.5.1-python3.10
USER root
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B7B3B788A8D3785C
RUN apt-get update
RUN apt-get install -y r-base r-recommended default-jre default-jdk
RUN R -e "install.packages('RJDBC', repos = 'http://cran.us.r-project.org')"
RUN R -e "install.packages('gtools', repos = 'http://cran.us.r-project.org')"
RUN R -e "install.packages('data.table', repos = 'http://cran.us.r-project.org')"
RUN R -e "install.packages('dplyr', repos = 'http://cran.us.r-project.org')"
RUN R -e "install.packages('ABCanalysis', repos = 'http://cran.us.r-project.org')"
RUN R -e "install.packages('forecast', repos = 'http://cran.us.r-project.org')"
RUN R -e "install.packages('reshape2', repos = 'http://cran.us.r-project.org')"
RUN R -e "install.packages('mefa', repos = 'http://cran.us.r-project.org')"
RUN R -e "install.packages('rJava', repos = 'http://cran.us.r-project.org')"
USER airflow

 docker build -t apache/airflow:2.5.1-python3.10-R .

Algorithms and Data Structures Matter

Number of IoT devices: 90k

Volume: 24h of measures

 

DB engines:

PostgreSQL:  1346,89 GB

InfluxDB: 92,16 GB

VictoriaMetrics: 19 GB

Clickhouse: 2.75 GB

Query of last measures for all devices:

VictoriaMetrics: 20-40s

Clickhouse: 1.5-2s

Hardware for Deep Learning

Hardware for Deep Learning.ToC

 Hardware for Deep Learning. Part 4: ASIC

Reference design for 8-20 DGX A100 POD Ethernet storage network

Compute network:

    QM8790 40xHDR200

Storage network:

    SN3700 32x100GbE/ SN4600 64x100GbE

InBand network:

    SN3700 32x100GbE / SN4600 64x100GbE / SN2010 18x25GbE+4x100GbE

OOB network:

    SN2201 48xGbE+4x100GbE

The Square Kilometre Array Science Data Processor Preliminary Compute Platform Design

The Square Kilometre Array Science Data Processor Preliminary Compute Platform Design