The Raspberry Pi Thread

[Endgame124]

Table of Contents and Change Log
Why use a Raspberry pi?
-Overview
-Pi Versions
-Resources / Links

Pi OS and installation.
-Preferred OS for Boinc only
-Other OS
-Tweaking Raspbian Lite for DC use

Pi Tweaks and Configuration.
-Shutting down non essential hardware
-Over / Under clocking
-Thermals and Heatsink
-Stretching Ram

Boinc Topics.
-Arm on Linux / Raspberry Pi Supported Projects
-Boinc installation on Raspbian
-Managing many Boinc Clients

Minimizing Costs for multi pi Boinc deployments.

Networking and Management.

Putting it all together (Pi Equipment / configurations).

Change Log
Version 1.2 - added additional power saving and OS trimming commands, and a pi cluster sample purchase list.
Versions 1.1 - added several links, a resources section, and some information under Putting it all together..
Version 1.0- multiple sections with minimal information

 

[Endgame124]

Why use a Raspberry pi?
Overview
I've been using 4GB Raspberry Pis for Rosetta, and they are quite energy efficient when compared to my other hardware. Additionally, for a minimal purchase, I can incrementally add more processing power.

Pi options
The Current version is the Pi 4, which comes with 2GB ($35), 4GB ($55), or 8GB ($75) of ram.
There is very little reason to get any of the older Pis as they are going to be roughly the same price as the 2GB Pi 4 and will offer much less performance.

Older Versions I've used
Pi 2B v1.1
900Mhz Quad core Cortex A7, rev1.1 is the last 32 bit only processor. I use it to run Pi hole and 3 processes of Asteroids@Home
Pi 3B
1.2GHz Quad Core Cortex A53, 1GB ram. I use these with Xbian, a HD Homerun, and a Cable Card to replace my cable boxes. I also run a mix of WCG, Einstein@Home, and Asteroids on them.
Pi 3B+
1.4GHz Quad Core Cortex A53, 1GB ram. I initially setup this one to process Rosetta, but ram was too big of a concern, so I switched it to run only WCG Open Pandemics.

Resources
Official Raspberry Pi page: https://www.raspberrypi.org/
Rosetta Forum Thread for the Pi 4: https://boinc.bakerlab.org/rosetta/forum_thread.php?id=13732

 

[Endgame124]

Pi OS and installation
My Preferred OS build for Boinc (Raspbian lite)
32 Bit Raspberry Pi OS (Raspbian)
64 Bit Raspberry Pi OS (Beta 64 bit build)

Other OS
Ubuntu - Ubuntu compiled for arm / Raspberry Pi. Runs heavier (uses more ram) than Raspbian, and does not have the nice Pi specific commands.
Xbian - Media center OS with Kodi. Boinc runs fine, but Xbian does not play nicely with 64 bit user space. This means you won’t be able to run Rosetta on it.

My Recommendations for Reducing the Size and Memory Ovehead of Raspbian Lite

#disable rsyslogd and replace with journalctl
sudo systemctl disable rsyslog.service
sudo sed -i "s/\#ForwardToSyslog=yes/ForwardToSyslog=no/" /etc/systemd/journald.conf
sudo mkdir /etc/logrotate.d/.save
sudo mv /etc/logrotate.d/rsyslog /etc/logrotate.d/.save

#disable microsoft apt repos
sudo rm /etc/apt/sources.list.d/vscode.list
sudo rm /etc/apt/trusted.gpg.d/microsoft.gpg
sudo apt update

#disable the triggerhappy hotkey daemon that controls things like keyboard volume control
sudo systemctl stop triggerhappy
sudo systemctl disable triggerhappy
sudo systemctl disable triggerhappy.socket

#Set current IP address as static, and disable and remove dhcpd, switch ssh from ssh.service to ssh.socket
###
#!!!Manual Step
###
vi /etc/network/interfaces.d/eth0
set gateway, netmask, and dns-nameservers for your network.
set address line to:
address IPADDR
###
#!!! end manual step
###

STATIC_IPADDR=`ip -f inet addr show eth0 | awk '/inet / {print $2}' | awk -F '/' '{print $1}'`
sudo sed -i "s/IPADDR/$STATIC_IPADDR/" /etc/network/interfaces.d/eth0
sudo systemctl stop dhcpcd
sudo systemctl disable dhcpcd
sudo systemctl stop avahi-daemon
sudo systemctl disable avahi-daemon
sudo systemctl disable ssh.service
sudo systemctl enable ssh.socket

#cleanup unneeded dhcp packages
sudo apt-get remove dhcpcd5 alsa* -y && sudo apt autoremove -y



Reducing the size of Pi OS Lite

Need help trimming Pi OS lite for Pi B - Raspberry Pi Forums

www.raspberrypi.org

Raspbian Lite-Er - Raspberry Pi Forums

www.raspberrypi.org

 

[Endgame124]

Pi Tweaks and Configuration

Shutting down non essential hardware
If you're only using a headless Raspberry Pi for distributed computing, you can shutdown quit a bit of the onboard features to save power. I have only tested the following in Raspbian Lite, but they should all work in any distribution of Raspbian. I also assume that you will be using wired networking - setting up dozens of wireless clients can cause Wifi interferance that I did not not want to deal with.

Many configurations involve modifying /boot/config.txt. Any line that has a reference to this file will require a reboot to take effect.

If a command does not involve /boot/config.txt, it will last until the next reboot. I personally created a small shell script called power.sh to turn off HDMI, the power LED, and USB.

#Disable Wifi, and disable check for wifi
echo "dtoverlay=disable-wifi" | sudo tee -a /boot/config.txt
sudo systemctl disable wpa_supplicant
sudo rm /etc/profile.d/wifi-check.sh

#Disable Bluetooth and bluetooth services
sudo systemctl disable hciuart
echo "dtoverlay=disable-bt" | sudo tee -a /boot/config.txt

#disable host controller serial transmitter (hciuart)
sudo systemctl disable hciuart

#Disable HDMI:
sudo tvservice --off

#Turn off Power LED:
echo 0 | sudo tee /sys/class/leds/led1/brightness

#disable network Power LEDs
echo "dtparam=eth_led0=4" | sudo tee -a /boot/config.txt
echo "dtparam=eth_led1=4" | sudo tee -a /boot/config.txt

#disable audio
sudo sed -i "s/dtparam=audio=on/dtparam=audio=off/" /boot/config.txt

Power Savings reference thread:

RPi4 - Power Savings - Raspberry Pi Forums

www.raspberrypi.org

Over / Under clocking
The ARM CPU can be over and under volted and over and under clocked the same as any other CPU, though there are limits baked into Raspbian.

Frequency settings are controlled in /boot/config.txt via the setting arm_freq. Default is 1500 for the Pi 4, maximum is 2147.

#sample line to over clock to 2000 mhz
arm_freq=2000

Voltage Settings are controlled in /boot/config.txt via the setting over_voltage. Settings are 1-6 (over volting), and -1 to -6 (under volting), and each +/- adds / subtracts .0250V. Default is 0, which is .8750V

#sample line to over volt 5 (.125V)
over_voltage=5

Across 4 Raspberry Pi 4s, my most successful results are:

Frequency / voltage
1500 / -2
1750 / 2
1900 / 4
2025 / 5
2147 / 6 (needs excellent cooling for 24x7 usage, and even then can generate failed WUs)

Thermals and Heatsink
Offical raspberry pi org thermal testing.

Thermal testing Raspberry Pi 4 - Raspberry Pi

Raspberry Pi 4 just got a lot cooler! The last four months of firmware updates have taken over half a watt out of idle power and nearly a watt out of fully loaded power

www.raspberrypi.org

Best heatsink I've used is the Flirc case:

https://flirc.tv/more/raspberry-pi-4-case

Pi 4 4GB in a Flirc Case, w/ 100% load from 4 Rosetta processes, ~20c ambient, Max Temps:
Passive w/ plastic top on 58c
Active w/ top on + 60mm fan blowing at front opposite network cable 42c

Passive w/ plastic top off 55c
Active w/ plastic top off + 60mm fan blowing at front opposite network cable 35c
Active w/ plastic top off + 60mm fan blowing at side opposite power cable 40c

Passive w/ plastic top off + a Thermaltake Volcano 6cu+ heatsink resting on top 51c
Active w/ plastic top off + heatsink + 60mm fan blowing at side opposite power cable 38c
Active w/ plastic top off + heatsink + 60mm fan blowing at front opposite network cable 35c


Stretching Ram
This is most applicable to the 2GB Pi 4, or a 4GB pi 4 trying to run 4 Rosetta processes.

Reduce ram allocated to CPU
By default, 64MB of memory is allocated to the GPU. This is unnecessary when running headless, so it can be reduced to the mimimum, freeing up memory for the OS.

#set the GPU to 16MB, the minimum amount of allocated ram
echo "gpu_mem=16" | sudo tee -a /boot/config.txt

Configure Zram
Zram creates a compressed, in memory swap partition that is used when all available memory runs out. Testing shows this to work very well up to a partition size = to total memory size to increase the amount of fast memory available to the pi. This is expecially useful when you randomly get 4 1.2+gb processes running at the same time.
sudo apt-get install zram-tools

sudo vi /etc/default/zramswap
PERCENTAGE=100

sudo zramswap stop
sudo zramswap start

The default compression is lzo - I have found lz4 to be slightly better performing, but will cover that in a later update.

 

[Endgame124]

Boinc Topics
Arm on Linux / Raspberry Pi Supported Projects

Choosing BOINC projects

boinc.berkeley.edu

-Amicable numbers
-Asteroids
-Einstein
-LHC
-Radioactive
-Rosetta
-Universe
-World Community Grid
-Yoyo

Boinc installation on Raspbian
For GUI boinc install:
sudo apt-get install xorg openbox
sudo apt-get install boinc-client boinc-manager

For textUI boinc install:
sudo apt-get install boinc-client boinctui

Boinc Additional Configuration, if using zram
Edit the Boinc global preferences to use more than 100% of system memory:
sudo vi /etc/boinc-client/global_prefs_override.xml

update xml to:
<global_preferences>
<max_ncpus_pct>100.000000</max_ncpus_pct>
<vm_max_used_pct>90.000000</vm_max_used_pct>
<ram_max_used_busy_pct>150.000000</ram_max_used_busy_pct>
<ram_max_used_idle_pct>150.000000</ram_max_used_idle_pct>
<cpu_usage_limit>100.000000</cpu_usage_limit>
</global_preferences>

If also running Rosetta, reduce amount of work kept locally, as boinc will assign many more tasks thatn the pi can process initially:
Add to the above global preferences
<work_buf_min_days>.5</work_buf_min_days>
<work_buf_additional_days>1</work_buf_additional_days>


Managing many Boinc Clients

Points and comparison hardware:

Dual Epyc (appx 240 PPD/W) :

Page 2 - Question - Zen 2, clock speed vs power consumption vs ppd sweet spot?

Page 2 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

forums.anandtech.com

Ryzen 3600 (Rosetta: appx 11,900 RAC / 991 per thread):

Page 2 - Question - Zen 2, clock speed vs power consumption vs ppd sweet spot?

Page 2 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

forums.anandtech.com

 

[Endgame124]

Minimizing Costs for multi pi Boinc deployments
Multi port USB Charger - Currently using RavPower 6 port, trying Anker 6 port USB charger
Multi Pack USB cords
Make your own Ethernet Cables
Skip the SD Card and boot from the network
12 Port Switch
Use a Stacking case (ex: https://www.amazon.com/GeeekPi-Raspberry-Cluster-Stackable-12-Layers/dp/B08FH3V6GV?ref_=ast_sto_dp)

 

[Endgame124]

Networking and Management
Network Boot

Network Booting The Pi 4

We’ve talked about PXE booting the Raspberry Pi 3B+, and then looked at the Raspberry Pi 4 as a desktop replacement. But there’s more! The Pi 4 sports a very useful new feature, the fla…

hackaday.com

Raspberry Pi PXE Boot - Network booting a Pi 4 without an SD card - Linuxhit

Raspberry Pi PXE Boot tutorial. We show how to boot a raspberry pi 4 with no SD card. We setup a PXE boot server using PXE, DHCP and TFTP to netboot a Pi 4.

linuxhit.com

Network Storage
Swap over NFS

Tip: Swap over nfs - Raspberry Pi Forums

www.raspberrypi.org

Management / kubernetes

Everything I know about Kubernetes I learned from a cluster of Raspberry Pis | Jeff Geerling

www.jeffgeerling.com

Kubernetes (K8s) Private Cloud with Raspberry Pi 4s

Deploying a Kubernetes cluster on a public cloud provider is easy, but what if you want a private bar...

dev.to

Power over Ethernet

Mounting iscsi targets

How to Setup iSCSI Server (Target) and Client (Initiator) on Debian 9

This guide will walk through a basic setup involving a simple iSCSI server (target) and client (initiator) both running Debian 9 (Stretch).

www.tecmint.com

 

[Endgame124]

Putting it all together (Pi Equipment and Configuration)

Least Expensive:
Parts list for the cheapest way to try out a Pi:
2GB Pi 4,
8GB+ micro SD card
USB A to C cable

Total cost:
$35 (if you already have the cable and the SD card)
$50 (if you need to get both a cable and a SD card).

Image Raspbian lite onto the SD card, plug the pi into a USB 3 port on your PC, and place the pi such that it sits in the airflow path of an exhaust fan.

Base:
4GB Pi 4
FLIRC Raspberry Pi 4 aluminum case
32GB Samsung Select Micro SD card
Official Raspberry Pi 4 power supply

Total Cost: $90

The WCG Pi Cluster
12x 2gb Pi 4s
12x Micro SD cards (32GB is the current value size as of this post)
12 USB A to C cables.
a 12 stack case (ex: https://www.amazon.com/GeeekPi-Raspberry-Cluster-Stackable-12-Layers/dp/B08FH3V6GV?ref_=ast_sto_dp )
16 port switch - we'll be running at 100mbit to save power, so you can go with whatever is on sale
2x 6 port USB Chargers (ex: https://www.amazon.com/gp/product/B00P936188/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 )

Approximate Cost: $714

 

[Fardringle]

This seems like an interesting project. The Pi 4 is definitely a lot more powerful than the older versions. A cost/power analysis seems worthwhile.

For example, how many complete Pi 4 builds would it take to equal the BOINC production of a few years old i5 or Ryzen PC, and what would be the difference in electricity usage and the difference in total cost (including the case, cables, boot SD card, and power source)...

 

[Endgame124]

This seems like an interesting project. The Pi 4 is definitely a lot more powerful than the older versions. A cost/power analysis seems worthwhile.

For example, how many complete Pi 4 builds would it take to equal the BOINC production of a few years old i5 or Ryzen PC, and what would be the difference in electricity usage and the difference in total cost (including the case, cables, boot SD card, and power source)...

I can actually answer that against a 2700x in Rosetta. Ill break it down later tonight (I don’t have the exact power draw of the Ryzen at the moment)

 

[Endgame124]

Some quick numbers below.

Ryzen 2700X w/ 32GB ram, Seasonic 750W Titanium PSU, 3 Drives (2 NVME, 1x Sata SSD) and 15 Boinc Processes and a 1080ti folding is pulling about 420W according to my UPS. Ballpark its at about 200W if I were to guess if I stopped folding. If I had all 16 threads dedicated to Rosetta, I should get about 16,500 average Credit.

My Core2 Quad 9650 uses apprx 120W and generates 3200 Credit

A single power optimized (hdmi off, usb off, wifi off, bluetooth off, led off, under volted) Pi 4 4GB at 1500mhz pulls 3 W. It generates appx 1100 Average Credit.

An over clocked Pi 4 4GB at 2025Mhz pulls 5W and generates appx 1350 Average Credit.

I would need 15 Power optimized pi 4s (45W) , or 12.2 (61W) Overclocked Pi 4s to get the same amount of credit in Rosetta as the 2700X, or 3 Pi 4s to generate more credit than the Core2 Quad.

I would deploy Pis in batches of 6, in setups like this:

6x raspberry pi 4 4GB ($55 ea):330
1x 5 pack 32GB Samsung select SD cards + 1 card: $37
1x 6 port Anker USB charger: $25
2x 3 pack 1 ft usb a to c cables: $8.99
1x 8 port Switch $19

If overclocking, I would add 6x Flirc Cases (15.95 each): 95.70

If going power optimized, I would use some of the stack of VGA ram sinks I have here, but you can get a 30 pack of Pi 4 heat sinks (enough for 6 pis) for $8.

Total for 6 overclocked pi 4s: 525
Total for 6 power optimized Pi 4s: 437

Total for 12 overclocked Pi 4s (61W. appx same credit as the 2700X) $1050
Total for 18 power optimized Pi 4s (54W. 3300 more credit than 2700X): $1311

 

[StefanR5R]

Forget about RAC for performance comparisons. It is a very long-term measure, reaching far into the past. (Past credit is counted with a factor of 1/2 per week which went past since credit was granted, supposedly.)

Furthermore, if credits at Rosetta@home is being looked at, the variation between workunits needs to be taken into account. To compare two hosts, run them for several days together so that they receive similar batches of workunits. Then gather the credit stats of these days (perhaps from boincstats). For an even better picture, determine not just PPD but also the variability of PPD of the results of each host (e.g., standard deviation or a similar measure of variability).

Third, while it is certainly possible to make a desktop computer with an 8-core 14 nm CPU pull 200 W at the wall when running Rosetta@home, it is not actually necessary.

Fourth, to determine power consumption while running Rosetta@home, take multiple readings, or have the power meter integrate the power consumption over the given time frame. This is again due to variability between workunits.

(In short, Rosetta@home is among the more difficult projects when it comes to performance measurements or efficiency measurements.)

Edit,
another open question is how much host throughput is taken away by Folding@home, whose GPU application appears to utilize processor caches or/ and memory controllers quite a bit.

 

[Endgame124]

Forget about RAC for performance comparisons. It is a very long-term measure, reaching far into the past. (Past credit is counted with a factor of 1/2 per week which went past since credit was granted, supposedly.)

Furthermore, if credits at Rosetta@home is being looked at, the variation between workunits needs to be taken into account. To compare two hosts, run them for several days together so that they receive similar batches of workunits. Then gather the credit stats of these days (perhaps from boincstats). For an even better picture, determine not just PPD but also the variability of PPD of the results of each host (e.g., standard deviation or a similar measure of variability).

I've actually been running Rosetta on some of these hosts long term to allow credit to stabilize.

Core2Quad, running nothing but Rosetta since 3/28: https://boinc.bakerlab.org/rosetta/show_host_detail.php?hostid=3931896

1500mhz Pi 4 4GB, running nothing but Rosetta since 4/23::

Computer 4215281

2050mhz Pi 4 4GB, running nothing but Rosetta since 4/29:

Computer 4269102

The 2700X is not ideal. It has F@H running on it, but my guess at credit probably is a decent guess - I would look to others for a better credit per watt measure for modern hardware.

I can certainly measure on another project as well. someone on the Hardforum is already doing that with a pi 4 on open pandemics: https://hardforum.com/threads/firing-the-boys-back-up.1997124/#post-1044613429

Edit

Fourth, to determine power consumption while running Rosetta@home, take multiple readings, or have the power meter integrate the power consumption over the given time frame. This is again due to variability between workunits.

This is actually surprisingly difficult with Raspberry Pis with my equipment. I'm using APC BR 1500 G UPS, but it only measures watts in whole numbers. A pi can vary 30% and still read the same wattage.

 

[Fardringle]

Since I asked for some comparisons, I figured I should contribute as well. I installed Linux on an old laptop with an i5-2430M CPU just to see what it does compared to the Pi4.

No results yet since I just installed it a few minutes ago, but the BOINC CPU benchmarks show that the Pi4 isn't far behind the i5-2430M..

Endgame's Pi4:

Computer 4269102

i5-2430M laptop:

Computer 4588304

 

[Fardringle]

I let the old laptop run Rosetta for about 52 hours (to finish the last tasks) and ended up with around 3700 points, making it probably around 1600 points per day. That's a little bit more than the overclocked Pi4, but not by much, so I guess it's a reasonably fair comparison to use for a non-Pi equivalent CPU. And the PI uses a lot less power, especially when considering all of the other components sucking up electricity in that laptop.

 

[Endgame124]

The pi 4 is definitely faster than my laptop with a old i3 370m.

While the newest hardware isn’t cut and dry vs a pi4, 7+ year old hardware seems like it can be replaced with pis for much better performance per watt.

 

[JWMiddleton]

Excellent post!! We are leaving for Florida in the morning and I'm taking my RPi4-B to connect to the TV in the VRBO condo we rented. So, when I get back I'll see about setting my Pi to BOINCing.

John

 

[Endgame124]

I added some links to the network section, and am trying to get a Pi 4 2GB to network boot. Cloning the drive to my freenas is proving to be problematic - rsync keeps getting stuck, so I can definitely say the process is not foolproof.

The advantage to fully network booting the Pis is the reduction in components, and, if you have your network speed sorted, its also generally faster than a flash drive. For this test, I'm also trying a pi4 2GB with just a small heatsink and a USB fan.

Finally, I took a long shot and contacted Seasonic support to see if they had any way to provide USB C power from one of their power supplies. If I could directly attach Pis to a power supply, it could push the per pi cost of power to most the cost of the cable.

If I can get it all working, I could be set for increasing the compute for a cluster of Pis for less than $40 per board.

 

[Endgame124]

A few updates.

First, Seasonic does not have usb c power directly useable via modular power adapter, but they did recommend a molex to usb cable that I didn’t know existed: https://www.coolerguys.com/products...v-power-connecter?_pos=4&_sid=ac5f71448&_ss=r

This means you could possibly attach 30 pis to a power supply - certainly not enough to justify buying a stand alone power supply to do it, but there may be something to be said for integrating Pis into an existing distributed computing box?

On the small heatsink front, it works great with a small amount of airflow. In fact it works so well, I don’t think there is a reason to use the FLIRC cases unless you wanted to use passive cooling.

The heatsink I used is an old Zalman ZM-RHS1 - its a super small VGA heatsink from like 10+ years ago. I have probably 30 of them on hand, as I bought them on clearance at something like $1 per box of 8 and only used a couple over the years.

Observations with the pi4 2GB in this configuration:

1) While in theory the 2GB model should use less power than the 4 GB model, my UPS shows the same 3-4W usage at 100% load. When set to over_voltage=-2, and with USB, HDMI, wifi, etc turned off, it reads a consistant 3W power usage, just like a 4GB model.

2) when turned on its edge (GPIO connector down, USB-C connector up), with the small heatsink and a fan blowing on it, it runs fine even over clocked at 2000 mhz and over volt 4 (max temp seen so far is 69C, and no throttling). I imagine that a more reasonably large heatsink designed for the pi4 would drop the temp in this scenario multiple degrees.

3) with a zram configuration for 100% (effective 4GB of ram), you can run 4 Rosetta processes most of the time. I suspect you could stretch it to 150% (effective 5GB ram) and you would almost always be running 4 Rosetta processes. That said, I think the ideal setup is to split the 2GB model between 2 Rosetta processes and 2 Open Pandemic processes.

 

[StefanR5R]

Are there any specifications given WRT accuracy of the power readings of the UPS, particularly at low Amperage? If not, assume a large relative error of measurement at this low level of power consumption.

 

[Endgame124]

Are there any specifications given WRT accuracy of the power readings of the UPS, particularly at low Amperage? If not, assume a large relative error of measurement at this low level of power consumption.

The product documentation does not specify any accuracy parameters to the wattage reading. I’ll have to contact APC directly, but I’m fairly certain it is not the best way to measure the power usage of the pi. That said, it is also the best way to measure the power usage available to me at this time.

 

[StefanR5R]

Seasonic does not have usb c power directly useable via modular power adapter, but they did recommend a molex to usb cable that I didn’t know existed: https://www.coolerguys.com/products...v-power-connecter?_pos=4&_sid=ac5f71448&_ss=r

This means you could possibly attach 30 pis to a power supply -

PC PSUs tend to make only a low fraction of their overall output Wattage available at their 5 V rail. Are you intending to use the PSU for a regular PC and let it power the Pis along with it? Otherwise, a dedicated large 5 V PSU would be good to have, but is perhaps hard to find.

Power over Ethernet would be nifty, but alas too costly due to the required DC/DC converters.
Edit, on the other hand, a PoE HAT for Raspberry Pi typically doubles as an active cooler, meaning you can save the cost of another passive or active heat sink.
Edit 2, the cheaper tier of PoE HATs or PoE splitters is probably best avoided. They seem to be compromised with inefficiency and other problems.

 

[Endgame124]

PC PSUs tend to make only a low fraction of their overall output Wattage available at their 5 V rail. Are you intending to use the PSU for a regular PC and let it power the Pis along with it? Otherwise, a dedicated large 5 V PSU would be good to have, but is perhaps hard to find.

Power over Ethernet would be nifty, but alas too costly due to the required DC/DC converters.
Edit, on the other hand, a PoE HAT for Raspberry Pi typically doubles as an active cooler, meaning you can save the cost of another passive or active heat sink.
Edit 2, the cheaper tier of PoE HATs or PoE splitters is probably best avoided. They seem to be compromised with inefficiency and other problems.

The best plan I’ve come up with so far would be to add Pi 4s to an existing pc - even 100 pi 4s would only use between 300 to 400 watts and I have yet to come up with a way to power even half that many from one power supply. I found some documentation about how to use the usb C cable for both power and data so you wouldn’t need Ethernet when attaching to a pc, but it requires a usb c port on the pc. C ports are not very economical to add to a pc.

On the PoE front, I found the same as you. The good ones are fairly expensive relative to the pi, and they add cost to the switch as well. If you want to have 5 or 6, it’s not a big deal. If we start talking about a deployment of 100+ pis, I think I would want to keep the cost as close to the cost of the pi as possible.

 

[StefanR5R]

For example, the 550...1200 W Platinum PSUs which I have here are rated for 22...25 A at the 5 V output, with a cap of 110...150 W of the 3.3 V and 5 V outputs combined. (This happens to be less than what the SSI EPS12V power supply design guide recommends.)

In addition, there is an independent 5 V standby power pin at the ATX connector, which is rated for 3.0 A.

 

[Endgame124]

Just had my first pi4 4gb go into swap thrash state, even with zram set to 4GB. It had 2 Rosetta processes at 2GB, 1 at 1.6gb, and one at 1.3gb. The compression ratio wasn’t great, and it ended up filling my 1gb disk swap partition.

If Rosetta is going to start assigning 2GB tasks regularly, I may have to start having 2 Rosetta tasks and 2 WCG tasks. That seems to annoy boinc Though, based on some testing with a 2GB pi.