Question Who is the Rasberry PI expert ? And a recommended system.

[Markfw]

I want to try to play with these. The Thor challenge got me thinking. I could get 10 of these, and it would be 40 cores ! What do you need to start ? I saw some ads on newegg, and it looks like you need more than just the board. The best star kit ? and for more, what else do you need for every one ?

What do you think of this ?

Are you a human?

www.newegg.com

@biodoc , @VirtualLarry , @lane42 Don't one of you have one of these ? Or know who does ? I am dying to buy one....

 

[Endgame124]

For multi pi case, I haven’t used it, but this one is recommended by someone on the hard forum

Amazon.com: GeeekPi Raspberry Pi Cluster Case, Raspberry Pi Rack Case Stackable Case with Cooling Fan 120mm RGB LED 5V Fan for Raspberry Pi 4B/3B+/3B/2B/B+ and Jetson Nano (8-Layers): Computers & Accessories

Amazon.com: GeeekPi Raspberry Pi Cluster Case, Raspberry Pi Rack Case Stackable Case with Cooling Fan 120mm RGB LED 5V Fan for Raspberry Pi 4B/3B+/3B/2B/B+ and Jetson Nano (8-Layers): Computers & Accessories

www.amazon.com

 

[Markfw]

Case is complicated. I’m using individual FLIRC cases, but I’ll share some ideas later.

Best place to buy multiple pis (cheaper than amazon)

American Raspberry Pi Shop

American Raspberry Pi Shop. One stop shop for all your Pi needs. Raspberry Pi add-ons, HATs, accessories, Starter Kit, Media Center kit, RetroPi Arcade kit, Ultimate Kit, and many other project kits. PiShop.us is approved Raspberry Pi re-seller and carries all official Raspberry Pi boards and...

www.pishop.us

Note, microcenter is cheapest for individual pi purchases, as they have a permanent $5 off MSRP In store option. However, you can only get one of each ram size per purchase with the discount (ex, buy a 2GB, 4gb, and 8gb in one purchase and get $15 off total).

OS: use raspbian lite. There are complications with this with Rosetta that are covered in my guide. I have shell scripts that will do all the config.

You’ll need a SD card reader / writer to flash the cards with the OS. Use the official software to do the flashing, it’s vastly simplified over any other process

OK, to stop half the question, I just ordered ONE to start with. Here: https://www.newegg.com/p/1HD-005N-000J9?Item=9SIAHUBBMR9923

I hope its a good setup. That site with multiple PI's looks like none of the "standard" ones.

Oh, and I like that case !!!

 

[Endgame124]

OK. The littlest one is in bed, and the older one is sitting next to me watching paw patrol, so I have some time to type.

On the kit you bought. These starter kits are generally designed as a quick start to a desktop replacement type system. The SD card comes preloaded with NOOBs - the full version of Raspbian, complete with linux desktop, and all the raspberry pi apps. Since ram is precious with the Pi, I run headless and do everything over SSH and manage from command line. My day job is also in IT as a Performance engineer with 20 years of systems engineering experience, so I'm completely OK running entirely from command line.

Note that Pis don't really power down like a normal computer - the best you can do is something like "shutdown now" from the command line and it will put the pi in deep sleep. You would need to unplug and replug to power back up. That is why the power supply has a switch on it - to facilitate turning it on and off as desired (not really a use case for my pis, which are running 24x7.

For other items in the bundle:

Case:
Its a clear, plastic case. It likely has little airflow (I bought a similar starter set with my first pi). These are fine when using the pi as a desktop usually at idle. For 100% cpu 24x7, you'll find yourself thermally throttling. k

My recommendation here: Don't attach the top half of the case, leave it open. Also, stand the pi on its side - there has been lots of testing done to show you'll knock a few degrees off a pi

SD card:
The Samsung evo is a good card. The samsung select that I linked is just the amazon exclusive of the same card that costs a little bit less. 32GB is plenty.

Heatsinks:
Really the only heatsink that matters is the one that you apply to the CPU (silver capped chip in the center-ish of the board). minimal airflow with a single heatsink is enough to keep the Pi from thermally throttling.

 

[Endgame124]

I've now read through the whole thread.

If I understand correctly, this is your goal:
Create a low power DC compute farm from 8 to 12 Pi 4s to use on Rosetta

Based on your questions and comments, I believe the secondary goal is:
Optimize purchases to support multiple Pis in a cost efficient manner

This goal is totally achievable!

I have approached my pi cluster, from a high level like this:

1) Optimize power usage per pi by tweaking the OS and under volting each individual pi. However, I only use settings that work on all Pi 4s, as I don't want to manage individual pis. I've been able to push power load down to about 4.25 - 4.5 watts per pi, which is about the same efficiency as a power optimized threadripper 3990 (as far as I can tell - I don't have a thread ripper to compare)

2) Optimize points per day by using the most cut down version of Raspbian (Debian linux tailored to raspberry pi) to save ram for DC. Additionally, I try to never mix projects on the same pi - setting 2 Pis to run 2 processes of open pandemics and 2 procesess of Rosetta each yields about 20% lower points / credit than than dedicating one pi to each project.

3) Optimize cost first by using different size pis for specific projects. Open pandemics easily fits within 1GB of ram, so I'm deploying half of my pis as 2GB pis. When open pandemics finishes, I'll pick other DC projects to support with the 2GB pis - hopefully it will be something else medical related.

4) Optimize cost second by smart purchasing additional components. Ex, the 6 port Anker power brick is dramatically less expensive than buying 6x individual power bricks. I've played around with just using a single heatsink on the pis (skipping cases all together), and network booting the Pis to avoid needing to buy SD cards. I've had mixed success with this - I can share if you're interested.

5) Optimize my time to manage this, meaning that I've scripted the setup, and have a few tweaks in place to be able to quickly identify problems with an individual pi.


For references, you have the link to my pi thread here on the AT forums. I just skimmed my pi thread and realize it needs a lot of work. Adjustments for the pandemic have meant that I just have not had time to put much work into that thread. Maybe I can do it over a few nights this week. Anyway...

I've been probably slightly more active on the hardforum. If you want to see my progression in working with Pis, you can read these two threads on the hard forum:

Firing the boys back up

Started folding again today. Right now I have a 3950x and 2080ti folding away and this next week I'll be bringing on another 14 cores with an i9-9940x. I picked folding because I could choose covid research. What projects are you guys running?

hardforum.com

Distributed Computing on Raspberry Pi

I figured it was time to start a Pi thread instead of hijacking someone else's thread with Pi information. Additionally, this should help people looking for information. I'm already working on a complete set of information at the AT forums (see references at the end of this post), but I...

hardforum.com

There is also a long thread on the Rosetta forums that may be an interesting read:

Running on a 4GB Raspberry Pi 4 - How to?

 

[Markfw]

For SD card, I prefer these

https://www.amazon.com/Samsung-MicroSDHC-Adapter-MB-ME32GA-AM/dp/B06XWN9Q99/ref=mp_s_a_1_7?dchild=1&keywords=samsung+select+32gb&qid=1600731091&sprefix=samsung+select+32&sr=8-7

Power loss to a pi while it’s using the card can cause damage. I’ve lost 3 of the super cheap microcenter cards. Have had no problems with these Samsung cards. That said, I’m now running them on UPS, which has resolved all the power issues.

Do you need that card all the time ? or just to boot ?

 

[Markfw]

And what do you think of this PSU ? 10 devices ?

https://www.amazon.com/gp/product/B016JZVZ8A/ref=ox_sc_act_title_1?smid=A7B2YWH9QZW4W&psc=1

OK, the rest of my cart. Please tell me what I am missing for an 8 card system (32 cores)

https://www.amazon.com/gp/product/B07TC2BK1X/ref=ox_sc_act_title_1?smid=AAU5UPIIBDRLP&psc=1

(times 7, only 4 gig ??)

Heat sinks ? (need one more)

Amazon.com: 30 PCS Raspberry Pi 4 Heatsink Kit with Thermal Conductive Adhesive Tape, Aluminum Heat Sink Cooler and Copper Heatsink for Raspberry Pi B B+ 2/3/4: Computers & Accessories

Buy 30 PCS Raspberry Pi 4 Heatsink Kit with Thermal Conductive Adhesive Tape, Aluminum Heat Sink Cooler and Copper Heatsink for Raspberry Pi B B+ 2/3/4: Heatsinks - Amazon.com ✓ FREE DELIVERY possible on eligible purchases

www.amazon.com

The case (you recommended it)

Amazon.com: GeeekPi Raspberry Pi Cluster Case, Raspberry Pi Rack Case Stackable Case with Cooling Fan 120mm RGB LED 5V Fan for Raspberry Pi 4B/3B+/3B/2B/B+ and Jetson Nano (8-Layers): Computers & Accessories

Amazon.com: GeeekPi Raspberry Pi Cluster Case, Raspberry Pi Rack Case Stackable Case with Cooling Fan 120mm RGB LED 5V Fan for Raspberry Pi 4B/3B+/3B/2B/B+ and Jetson Nano (8-Layers): Computers & Accessories

www.amazon.com

The 32 gig cards (you recommended it)

https://www.amazon.com/gp/product/B06XWN9Q99/ref=ox_sc_act_title_4?smid=ATVPDKIKX0DER&psc=1

Anything I am missing for the entire array ?

 

[Endgame124]

Do you need that card all the time ? or just to boot ?

If you boot from it, you need to leave it in(no hot swap on the sd card)

You need storage for a few things

1) A place to store the Boinc files and work units
2) something to boot from
3) swap

Assuming you have network storage, 1 is easy. 2 and 3 are a lot more work, but can be done if you want to run entirely without a SD card

 

[Markfw]

If you boot from it, you need to leave it in(no hot swap on the sd card)

You need storage for a few things

1) A place to store the Boinc files and work units
2) something to boot from
3) swap

Assuming you have network storage, 1 is easy. 2 and 3 are a lot more work, but can be done if you want to run entirely without a SD card

See above... BTW, so I need that adapter for the SD card that shows up in my newegg combo ? and will 4 gig work for WCG ? I think I will leave Rosetta for the EPYC boxes.

And lastly, what was the PPD/watt for EPYC vs the PI ????? I looked at the links, but its not clear.

 

[Endgame124]

posting while wrangling kids

Short answers

Power supply - no idea, I’ve found quality varies a lot per unit. A overclocked pi can take 6w, an under volted one as little as 4.25w. Depending on use case 60w may not be enough. No idea how efficient that supply is either - might be 60% or 85%.

Heat sink should be fine - I’m using old vga memory heart sinks and they work fine

The pi uses micro SD. The cards come with a SD adapter so you can use a SD writer with them. I have one on my laptop - if you don’t have one, you’ll need one

 

[Markfw]

BTW, the entire 32 core 8 slot PI setup mostly 4 gig, will be about $700 total. Thats why I am interested in it ppd VS EPYC. I have a 32 core EPYC (64 threads) sitting idle, as the motherboard is over $600 and memory is another $600

 

[Endgame124]

Answering a few more questions quick

WCG projects are all different from what I understand. I’ve only used open pandemics, and it uses less than 250mb per process, so you can used 2GB pi 4s without any issue.

On the SD card, the Samsung select cards come packaged with an adapter. The question is, do you have a sad writer?

 

[Markfw]

Answering a few more questions quick

WCG projects are all different from what I understand. I’ve only used open pandemics, and it uses less than 250mb per process, so you can used 2GB pi 4s without any issue.

On the SD card, the Samsung select cards come packaged with an adapter. The question is, do you have a sad writer?

I have many adapters that work with SD cardss to put them in USB slots, and I can transfer data that way. Is that what you mean ?

 

[Endgame124]

Open pandemics points per day varies a bit - here is a copy paste for one of my pis:

23/2020 0:004:07:08:38 6,424 11
09/22/2020 0:003:09:33:53 5,132 9
09/21/2020 0:003:18:42:07 5,665 10
09/20/2020 0:004:05:20:05 6,180 11
09/19/2020 0:003:13:18:22 5,165 9
09/18/2020 0:002:15:34:50 4,012 6
09/17/2020 0:001:19:38:27 2,680 5
09/16/2020 0:002:23:37:39 4,388 8
09/15/2020 0:003:10:32:08 5,056 9
09/14/2020 0:003:11:59:22 5,191 9
09/13/2020 0:004:04:44:26 6,265 11
09/12/2020 0:004:19:47:34 7,353 12
09/11/2020 0:003:00:26:19 4,720 8
09/10/2020 0:004:01:33:50 6,249 11

Eyeball says it is around 5,500 / day (I didn’t actually do the math). Power draw for the pi is around 4.5w with wired networking set to 100mbit, LEDs off, hdmi off, Bluetooth off, etc.

I don’t have good numbers for pandemics points per watt for the higher end systems.

Rosetta average credit for a pi 4 4gb varies between 1000 and 1300.

 

[Endgame124]

I have many adapters that work with SD cardss to put them in USB slots, and I can transfer data that way. Is that what you mean ?

I think so. I’m talking something like this:

Amazon.com: UGREEN SD Card Reader USB 3.0 Card Hub Adapter 5Gbps Read 4 Cards Simultaneously CF, CFI, TF, SDXC, SDHC, SD, MMC, Micro SDXC, Micro SD, Micro SDHC, MS, UHS-I for Windows, Mac, Linux: Computers & Accessories

Buy UGREEN SD Card Reader USB 3.0 Card Hub Adapter 5Gbps Read 4 Cards Simultaneously CF, CFI, TF, SDXC, SDHC, SD, MMC, Micro SDXC, Micro SD, Micro SDHC, MS, UHS-I for Windows, Mac, Linux: Memory Card Adapters - Amazon.com ✓ FREE DELIVERY possible on eligible purchases

www.amazon.com

I use the one built into my laptop, so this is an area I have little expertise with.

 

[Markfw]

I can't link to that power supply anymore, the link now says it can't find the page.

 

[Markfw]

Open pandemics points per day varies a bit - here is a copy paste for one of my pis:

23/2020 0:004:07:08:38 6,424 11
09/22/2020 0:003:09:33:53 5,132 9
09/21/2020 0:003:18:42:07 5,665 10
09/20/2020 0:004:05:20:05 6,180 11
09/19/2020 0:003:13:18:22 5,165 9
09/18/2020 0:002:15:34:50 4,012 6
09/17/2020 0:001:19:38:27 2,680 5
09/16/2020 0:002:23:37:39 4,388 8
09/15/2020 0:003:10:32:08 5,056 9
09/14/2020 0:003:11:59:22 5,191 9
09/13/2020 0:004:04:44:26 6,265 11
09/12/2020 0:004:19:47:34 7,353 12
09/11/2020 0:003:00:26:19 4,720 8
09/10/2020 0:004:01:33:50 6,249 11

Eyeball says it is around 5,500 / day (I didn’t actually do the math). Power draw for the pi is around 4.5w with wired networking set to 100mbit, LEDs off, hdmi off, Bluetooth off, etc.

I don’t have good numbers for pandemics points per watt for the higher end systems.

Rosetta average credit for a pi 4 4gb varies between 1000 and 1300.

I can't equate that to this, can you help ?

World Community Grid

What if you could support causes you care about while reading this post? Your device’s unused computing power can help scientists tackle some of the world’s biggest problems: from finding better #HIV and #cancer treatments to more efficient sources of renewable energy? I just joined World...

www.worldcommunitygrid.org

If that link does not work, can you look at my results ? Or compare to something you have ?

 

[Endgame124]

I can't equate that to this, can you help ?

World Community Grid

What if you could support causes you care about while reading this post? Your device’s unused computing power can help scientists tackle some of the world’s biggest problems: from finding better #HIV and #cancer treatments to more efficient sources of renewable energy? I just joined World...

www.worldcommunitygrid.org

If that link does not work, can you look at my results ? Or compare to something you have ?

I can’t see your individual devices it seems, but I can see your user statistics. You have an average points per hour of run time of 195, while mine is 48. Given how many devices you have, and the fact I have a mix of pi 2s, 3s, and 4s. that may not be a very useful comparison.

Here is an older screen shot of the open pandemics performance of one of my 2GB pis:

https://hardforum.com/data/attachment-files/2020/07/364262_9103F5A7-8F5A-44BF-8BDB-1CB1335D8939.jpeg

Perhaps that will be useful to you? If not, I can break it down to what ever formula you would like.

Ex, I could total it up, divide by 4 to get points per thread, and then divide by 14 to get average points per day per thread. That could be divided by 24 to get average points per hour per thread. Each thread is about 1.06w on the pi, so that could be converted to average points per hour per watt hour.

 

[Markfw]

I can’t see your individual devices it seems, but I can see your user statistics. You have an average points per hour of run time of 195, while mine is 48. Given how many devices you have, and the fact I have a mix of pi 2s, 3s, and 4s. that may not be a very useful comparison.

Here is an older screen shot of the open pandemics performance of one of my 2GB pis:

https://hardforum.com/data/attachment-files/2020/07/364262_9103F5A7-8F5A-44BF-8BDB-1CB1335D8939.jpeg

Perhaps that will be useful to you? If not, I can break it down to what ever formula you would like.

Ex, I could total it up, divide by 4 to get points per thread, and then divide by 14 to get average points per day per thread. That could be divided by 24 to get average points per hour per thread. Each thread is about 1.06w on the pi, so that could be converted to average points per hour per watt hour.

Whatever the best you can do. I am just trying to figure out if its worth the $700 vs another 16 or 32 core EPYC. I even have one if I can find the motherboard cheap.

 

[Endgame124]

Whatever the best you can do. I am just trying to figure out if its worth the $700 vs another 16 or 32 core EPYC. I even have one if I can find the motherboard cheap.

From my WCG stats, where each Pi 4 is a 2GB variant, undervolted and power tweaked, using approximately 4.25 watts to 4.5 watts. Huge note here: each pi has a small variance in voltage and power draw from my experience, especially when going from 2GB to 4GB to 8GB models.

Additionally I don't 100% trust my UPS and Power meter at these low power draws, so there is some uncertainty. The_Heretic over on the [H] forum cut up a power supply and did some actual readings with a volt meter, and got 4.45w for a stock 4 GB Pi 4, which would put a power optimized one right around, or just under, 4w.

Power supplies also play a large part in the efficiency, and these usb power bricks are very hard to get efficiency numbers on. I e-mailed multiple vendors, and only Anker got back to me, stating that their power supplies are 85% efficient at any load. Real world testing only shows that 85% is when the power brick is near 50% capacity - with a very small load from a single pi, its more like 70%.

Averages for 1 pi:
4.25-4.5 watt drawn at the wall
Daily: 5263 points at 102 - 108 watt hours.
Hourly (real world): 219.29 points at 4.25 - 4.5 watt hours.
Per thread Per hour: 54.82 points at 1.0625 - 1.125 watt hours

expected for 8 pis.
34-36 watt drawn at the wall
Daily: 42,104 points at 816 - 864 watt hours.
Hourly (real world): 1,754.33 points at 34 - 36 watt hours.

There may be some variance between compute hours and real world hours as listed in the WCG stats, but I've run out of time to work out the actual compute time of the stats - it should be fairly close to real world hours, given its averaged over 13 days per pi.

My approximation of WCG OP Points per day per watt: 51.59

In past comparisons, the Pi 4s have come out slightly less efficient than a power optimized 3950x, and more efficient than a stock 3950x. Those big Ep

 

[Markfw]

@Endgame124
OK, it came today. I installed everything like the manual says. I plug in the power, and I get a multicolored screen. It says I am supposed to click install, but there is nothing on the screen, not even a mouse cursor.

 

[Endgame124]

@Endgame124
OK, it came today. I installed everything like the manual says. I plug in the power, and I get a multicolored screen. It says I am supposed to click install, but there is nothing on the screen, not even a mouse cursor.

Something like this thread?

Multi coloured rainbow screen problem - Raspberry Pi Forums

www.raspberrypi.org

I’ve heard of the rainbow screen and it being related to insufficient power. I don’t even own a mini hdmi cable, so I’ve never actually used a pi 4 with a screen before. I’ve always used raspbian lite and done everything via ssh and command line.

 

[Markfw]

Well, they said something about the red light flashing meant bad power, but I have no flashing.. Solid red.

 

[Endgame124]

Per this thread, hdmi 0 is the default. If your plugged into hdmi1, it will just show rainbow. Have you tried the other hdmi port?

Raspberry Pi 4 Rainbow Screen - Raspberry Pi Forums

www.raspberrypi.org

 

[Markfw]

Per this thread, hdmi 0 is the default. If your plugged into hdmi1, it will just show rainbow. Have you tried the other hdmi port?

Raspberry Pi 4 Rainbow Screen - Raspberry Pi Forums

www.raspberrypi.org

I did not see that, but now I have a real screen using the other port !

 

[Markfw]

update. I now I have OS installed, but the update failed for some reason (OS updates). I have rebooted. It was probably due to me uplugging the network cable. Its now on wifi. How do I start that again ?