savannah-hackers-public
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Savannah-hackers-public] "Tokyo datacenter" for FSF


From: Ian Kelling
Subject: Re: [Savannah-hackers-public] "Tokyo datacenter" for FSF
Date: Sat, 20 Jul 2024 07:38:45 -0400
User-agent: mu4e 1.10.3; emacs 31.0.50

Thank you so much for the offer. We are currently in need of a DNS
server, so maybe you can help us there. I'm about to talk to you in irc


Jing Luo via "Discussions among Savannah Hackers, open subscription" 
<savannah-hackers-public@gnu.org> writes:

> [[PGP Signed Part:Undecided]]
> Hello Ian, Michael, Bob, Corwin, Amin,
>
> First of all, my condolences to Ian's and Michael's livers. No one should 
> have to work an all-nighter, so that's why I'm here hoping I can help.
>
> As I proposed on IRC, I can offer my VMs (& my expertise) to FSF so that 
> hopefully they can provide fail-over or load-balancing. I have a cluster of 
> servers that consists of various different spec, some are freedom-friendly 
> and some are less so. So here is a detailed list of machines I own that can 
> host VMs and each of their pros and cons (in my opinion) for your 
> consideration (also for future documenting purpose). Please let me know what 
> you think and what they can provide for FSF (e.g. DNS, MariaDB cluster, web 
> etc.). Suggestions are very welcome.
>
> Geolocation: Tokyo, Japan
> Electricity: 100% renewable energy.
> Internet (not a machine but worth mentioning):
> ISP1:
>  10Gbps residential (unshared), unmetered and unlimited, 1 static IPv4, a /56 
> block of IPv6.
>  Pros: high speed and unmetered. Serving 5TB/day on average now, peak traffic 
> 55TB/day.
>  Cons: the ISP doesn't support rDNS.
> ISP2:
>  1Gbps semi-business-residential (fiber shared with the apartment building), 
> data caps at 1TB per month, a /28 block of IPv4 and a /56 block of IPv6.
>  Pros: the ISP supports rDNS with IPv4. Suitable for name servers.
>  Cons: speed severely throttles after 1TB/month.
>
> Hostname: h12ssl-nt
> CPU: AMD EPYC 75F3, 32 cores
> RAM: 8x64=512GB DDR4 ECC RDIMM
> Motherboard: Supermicro H12SSL-NT
> Storage:
>  boot drives: 2x 1.6TB Kioxia CM6-V, PCIe Gen 4 NVMe SSD, ZFS RAID1, with an 
> Intel Optane P1600X 58GB, PCIe Gen3 NVMe SSD as SLOG
>  general purpose storage: 10x 7.68TB Kioxia CD6-R, PCIe Gen 4 NVNe SSD, ZFS 
> RAIDz2, with an Intel Optane P1600X 58GB, PCIe Gen3 NVMe SSD as SLOG
> Description:
>  This server runs an installation of Proxmox VE in an ATX PC case. Most of 
> the components were bought used from the second hand market, except for the 
> two Intel Optane SSDs. It provides many services important to the community 
> and essential to my digital life, such as:
>   - 1 node of repo.jing.rocks (the largest free software mirror in Japan, no 
> kidding)
>   - 1 authoritative name server and 2 recursive DNS servers (pihole, with 
> DHCP)
>   - 1 MariaDB cluster node, 2 Postgresql cluster nodes for various services
>   - 1 node of invidious.jing.rocks (luckily has not been banned by google)
>   - mastodon.jing.lgbt (1 node for load-balancing)
>   - multiple web servers in LXC containers
>   - 1 Proxmox Mail Gateway cluster node
>  It's also worth mentioning that I provide {web,name,mail} servers for 
> Dragora GNU/Linux-libre.
> Pros:
>  It's very stable, high performance, is also used as a NAS and a build 
> machine.
> Cons:
>  Has the AMD equivalent of Intel ME (forgot the name...). Has onboard 
> non-free IPMI but not connected to the internet. Built this machine when I 
> didn't really know free software. Also it's at risk of running out of RAM 
> because of ZFS filesystem caching (which Bob disagrees)
>
> Hostname: rome2d16-2t
> CPU: 2 sockets: AMD EPYC 7773X, 2x64=128 cores
> RAM: 16x64=1024GB DDR4 ECC RDIMM
> Motherboard: Asrock ROME2D16-2T
> Storage:
>  boot drives: 2x 1.6TB Kioxia CM6-V, PCIe Gen 4 NVMe SSD, ZFS RAID1, with an 
> Intel Optane P1600X 118GB, PCIe Gen3 NVMe SSD as SLOG (shared)
>  general purpose storage:
>   - 11x 16TB SATA spinning hard drives, ZFS RAIDz3, with an L2ARC of 3.84TB 
> SATA SSD, with a three-way mirrored special metadata vdev 1.92TB, with an 
> Intel Optane P1600X 118GB, PCIe Gen3 NVMe SSD as SLOG (shared)
>   - 11x 18TB SATA spinning hard drives, ZFS RAIDz3, with an L2ARC of 3.84TB 
> SATA SSD, with a three-way mirrored special metadata vdev 1.92TB, with an 
> Intel Optane P1600X 118GB, PCIe Gen3 NVMe SSD as SLOG (shared)
>   - 2x 8TB SATA spinning hard drives, ZFS RAID1, with an Intel Optane P1600X 
> 118GB, PCIe Gen3 NVMe SSD as SLOG (shared)
>   - 2x 3.84TB SATA SSD, ZFS RAID1, with an Intel Optane P1600X 118GB, PCIe 
> Gen3 NVMe SSD as SLOG (shared), reserved for gcc compile farm
> Description:
>  This server runs an installation of Proxmox VE in an EATX PC case. Many 
> components were bought used. It mainly provides these services:
>   - 3 VMs for gcc compile farm [1]: {cfarm420,cfarm421,cfarm422}.cfarm.net (I 
> specifically asked for those host names :)
>   - 1 node of repo.jing.rocks
>   - 1 authoritative name server and 1 internal DNS server (pihole, with DHCP)
>   - 1 vcs server that runs a forgejo instance and a savane instance (testing 
> only)
>   - 1 MariaDB cluster node, 2 Postgresql cluster nodes for various services
>   - mastodon.jing.lgbt (1 node for load-balancing)
>   - 1 Proxmox Mail Gateway cluster node
> Pros:
>  Suitable for highly parallel workload. The second ZFS pool has about 50TB of 
> space available, while others are at about 75~80% capacity.
> Cons:
>  Has the AMD equivalent of Intel ME (forgot the name...). Has onboard 
> non-free IPMI but not connected to the internet. Built this machine when I 
> didn't really know free software. Also it's at risk of running out of RAM 
> because of ZFS filesystem caching (which Bob disagrees).
>
> [1] https://portal.cfarm.net/machines/list/
>
> Hostname: z490
> CPU: Intel Core i9-10900K
> RAM: 4x32=128GB DDR4 UDIMM
> Description:
>  It also runs Proxmox VE, but mainly for more "not highly parallel" services, 
> like jitsi, nextcloud, invidious. About to be turned into a second-level L3 
> switch. It has Intel ME.
>
> Hostname: x570d4u-2l2t
> CPU: AMD Ryzen 9 5950X, 32 cores.
> RAM: 4x32=128GB DDR4 ECC UDIMM
> Motherboard: Asrock X570D4U-2L2T
> Description:
>  It also runs Proxmox VE, but mainly functions as a router/core switch. It's 
> super stable, has a long uptime. It has 10 10GbE ports in total. It runs 2 
> OpenWRT VMs, 1 ZNC IRC bouncer, 1 Proxmox Mail Gateway cluster node, and 1 
> reverse proxy/load-balancer for all web servers. Has a 4x960GB SATA SSD ZFS 
> RAID0 that isn't storing anything...
> Pros: It boots really fast.
> Cons:
>  Has the AMD equivalent of Intel ME (forgot the name...). Has onboard 
> non-free IPMI but not connected to the internet. The boot drives are about to 
> be replaced with ZFS RAID1, so a reinstall is coming.
>
> Hostname: altrad8ud-1l2t
> CPU: Ampere Altra Max Q128-30 engineering sample, 128 cores.
> RAM: 8x16=128GB for now, upgrading to 512GB
> Motherboard: Asrock ALTRAD8UD-1L2T
> Description:
>  A new machine that is not x86! It runs a version of Proxmox VE that I 
> patched and built from source then ported to arm64. It runs Trisquel 
> perfectly in my testing. Currently not running any service. About to be 
> reinstalled, replacing btrfs raid1 with zfs raid1.
> Pros: 100% free. Suitable for highly parallel workload, preparing to offer 
> VMs to Trisquel build farm and gcc compile farm. Runs really cool.
> Cons: Asrock ships a non-free UEFI/BIOS and a non-free distribution of 
> OpenBMC (not connected to internet).
>
> Hostname: (none)
> CPU: Rockchip RK3588
> RAM: 32GB LPDDR4
> Storage: 4TB NVMe SSD, consumer grade, and onboard eMMC and/or microSD cards.
> Description:
>  I also have 3 rockchip rk3588 based SBCs. They have my patched Proxmox VE 
> arm64 installed, with a custom kernel build. They all can run linux-libre 
> without framebuffer support (I gave up on those a long time ago). They are 
> under powered for compiling jobs, but suitable for server use.
> Pros: Stable and power efficient.
> Cons:
>  Has a fatal flaw that requires a non-free blob to boot. The DDR init blob 
> must be inserted into u-boot, or else it doesn't boot. Currently using a 
> custom u-boot build. I hope maybe someone can reverse engineer it someday...
>
> Some useful URLs:
> https://stats.jing.rocks/
> https://munin.jing.rocks/+(add "munin" or "munin/", want to avoid scraping 
> bots here)
> https://git.jing.rocks/cgit/home-config.git/
> https://goaccess.jing.rocks/
>
> Thanks for reading, looking forward to your opinions. I'm going back to patch 
> proxmox and linux...




reply via email to

[Prev in Thread] Current Thread [Next in Thread]