Questions on Homelab setup


Userlevel 7
Badge +9

I wish to build a new HOMELAB. Here is a guide and many others I found on Reddit.
- How is your lab setup? What would you recommend? Main virtualisation solution would be VMware (would need multiple hosts here). I would also have HyperV and Proxmox VE within this environment. Yes, this is possible!

I have got my ideas, but would like to learn from you. Keep in mind, I need a cost effective solution ranging from hardware to Power etc. Here is a talking point but just talking about the installation. I do NOT need help with the setup (deployment) tips as this is a walkthrough for me. Just ideas on how to setup low a cost effective lab, but yet a Lab to be reckoned with.

  •  


62 comments

Userlevel 7
Badge +8

Hey on this topic. What about these NUC things? i.e. very small, low power. Any suggestions?

Im testing a “mini” computers, they look great!
Better than a raspberry pi, but a bit limited.

https://lfconsulting.org/zima-board

Is in Spanish, but here the official website in English!

https://www.zimaboard.com

you’ve got different versions, it has 2 nics and, 2 sata 3, and 1 pic-e x4 port!

amazing!

cheers.

Thanks @HunterLAFR I need something small that passes the Burke Customs and Border control for tech stuff in the house, so it has to be small and not noticeable :) . I wanted to get on the Pi stuff but they were hard to get at one point and pretty expensive. However, I might revisit that. 

Picture comparing both

pi on the left!

 

Userlevel 7
Badge +22

Hey on this topic. What about these NUC things? i.e. very small, low power. Any suggestions?

Im testing a “mini” computers, they look great!
Better than a raspberry pi, but a bit limited.

https://lfconsulting.org/zima-board

Is in Spanish, but here the official website in English!

https://www.zimaboard.com

you’ve got different versions, it has 2 nics and, 2 sata 3, and 1 pic-e x4 port!

amazing!

cheers.

Thanks @HunterLAFR I need something small that passes the Burke Customs and Border control for tech stuff in the house, so it has to be small and not noticeable :) . I wanted to get on the Pi stuff but they were hard to get at one point and pretty expensive. However, I might revisit that. 

Userlevel 7
Badge +17

I personally haven’t had a lab in years. Back in the day, literally all I used was VMW Workstation config’d with 4-5 VMs, which did me well for what I was using it for. I need to think about actually investing in a real one...or, think about what my career forcus will be moving forward, meaning I may not even need one. 

Really great thread of comments in this post all! Lots to digest!

Userlevel 7
Badge +8

Hey on this topic. What about these NUC things? i.e. very small, low power. Any suggestions?

Im testing a “mini” computers, they look great!
Better than a raspberry pi, but a bit limited.

https://lfconsulting.org/zima-board

Is in Spanish, but here the official website in English!

https://www.zimaboard.com

you’ve got different versions, it has 2 nics and, 2 sata 3, and 1 pic-e x4 port!

amazing!

cheers.

Userlevel 7
Badge +22

Hey on this topic. What about these NUC things? i.e. very small, low power. Any suggestions?

Userlevel 7
Badge +8

Holy Guacomole! you have a huge basement and great lab. My basement is all done and my wife said no way to anything technical so I have to hide laptops in different places. I had one successfully acting as a server on the floor until…. my cat Carlos decided to chase after a spider down there… :(

 

Actually, it’s my garage.  Basement is not as large, but I can get 4 cars in my garage and still have some space for a shop/storage area.  Unfortunately, after 15 years, it’s near filled with crap and needs a massive cleaning.

That’s huge bro!

inhave a small warehouse, but as I also read somewhere, the wife is tracing me to reduce the lab day by day!

and my Diogenes IT syndrome it’s being active many years!

cheers!

Userlevel 7
Badge +8

I’m lucky we have so much hydro power here. 1 rate, all year, all hours. never changes.  It’s hard to calculate when different hours are busier than others, or seasons.

 

Also I can look around but threes a good chance I have some extra ram kicking around somewhere too. 

 

I just checked and I have 128GB of Hynix HMA82GR7MFR4N-UH 16GB DDR4-2400MHz PC4-19200 ECC Registered CL17 288-Pin DIMM 1.2V Single Rank Memory Module.   Too bad that won’t work in a G8. 

😭
thanks buddy!

had to try.

Userlevel 7
Badge +8

Lucky guys!
Im setting up my lab with super low power / consumption hardware, 
the electricity bill is killing me here in Spain!!

Does anyone have spare ram for a Proliant Gen 8??
128GB extra would be soooo nice!

🤣🤣🤣

cheers!

🤙

 

I might be able to come up with some RAM for you Luis.  I have a couple of DL180 G8 or G9’s that are being retired…..I just need to go pick them up since I just replaced them with PowerEdge R440’s a couple months ago.

Ohhhhhhhhhh

😍

ill take your word!

one day!

thanks!

Userlevel 7
Badge +6

Holy Guacomole! you have a huge basement and great lab. My basement is all done and my wife said no way to anything technical so I have to hide laptops in different places. I had one successfully acting as a server on the floor until…. my cat Carlos decided to chase after a spider down there… :(

 

Actually, it’s my garage.  Basement is not as large, but I can get 4 cars in my garage and still have some space for a shop/storage area.  Unfortunately, after 15 years, it’s near filled with crap and needs a massive cleaning.

Userlevel 7
Badge +22

Funny how this came up...last weekend I finally got around to installing a 20amp circuit in my garage so I could use a Tripp-Lite 2000VA UPS that I had laying around and got most of my gear plugged into it.  I had planned on sharing, so now seems as good time as any to post some pictures.  The homelab is pretty crude as it sits on top of a refrigerator in my garage and is subjected to extreme hot and extreme cold temperatures and humidity (and lack thereof).  But it gets the job done and really this hardware is solid.  The R610 that this R520 replaced ran for 3 or 4 years after it was retired from my client’s site, so in all, it was finally about 10 or 11 years old when it let out the magic smoke. 

Please don’t mind the disaster that is my garage…..I’ve learned that if you have a lot of space, you keep a lot of crap.

 

Homelab on top of the garage fridge.  Tripp-Lite 2000VA UPS, Dell PowerEdge R520 running VMware ESXi 8.0, Extreme Summit 1Gb POE 48-port switch, Ubiquiti Unified Security Gateway Pro, Synology 2-bay NAS connected via NFS for primary Veeam repository.  The CenturyLink Zyxel DSL Modem/Fiber router is a cold spare in case the Ubiquiti USG fails.  Onkyo receiver on top of everything is used to power some outdoor speakers in the back yard and indoor speakers in the garage and audio signal is supplied by a Google ChromeCast audio.

 

Side-view of the home lab with VMware ESXi 8.0 console on the monitor.

 

 

Rear view of the homelab equipment with a Geist PDU and and the newly installed 20amp circuit on the ceiling to power the UPS.

 

Holy Guacomole! you have a huge basement and great lab. My basement is all done and my wife said no way to anything technical so I have to hide laptops in different places. I had one successfully acting as a server on the floor until…. my cat Carlos decided to chase after a spider down there… :(

Userlevel 7
Badge +8

I’m lucky we have so much hydro power here. 1 rate, all year, all hours. never changes.  It’s hard to calculate when different hours are busier than others, or seasons.

 

Also I can look around but threes a good chance I have some extra ram kicking around somewhere too. 

 

I just checked and I have 128GB of Hynix HMA82GR7MFR4N-UH 16GB DDR4-2400MHz PC4-19200 ECC Registered CL17 288-Pin DIMM 1.2V Single Rank Memory Module.   Too bad that won’t work in a G8. 

Userlevel 7
Badge +6

Lucky guys!
Im setting up my lab with super low power / consumption hardware, 
the electricity bill is killing me here in Spain!!

Does anyone have spare ram for a Proliant Gen 8??
128GB extra would be soooo nice!

🤣🤣🤣

cheers!

🤙

 

I might be able to come up with some RAM for you Luis.  I have a couple of DL180 G8 or G9’s that are being retired…..I just need to go pick them up since I just replaced them with PowerEdge R440’s a couple months ago.

Userlevel 7
Badge +6

Lucky guys!
Im setting up my lab with super low power / consumption hardware, 
the electricity bill is killing me here in Spain!!

Does anyone have spare ram for a Proliant Gen 8??
128GB extra would be soooo nice!

🤣🤣🤣

cheers!

🤙

How much is power there? It’s about .09/kWh where i live. in CAD.  so $0.07 USD roughly.

 

Plus we can use the heat in the winter to help heat our houses hahahaha

 

Oh wow...that’s pretty cheap...mine is more expensive - at least in the summer months, this is a lower rate for me because I have SEER 14 heat pump - people with less efficient cooling have to pay a couple pennies more per kWh.  I should be paying .0936/kWh in the summer, and .0484/kWh in the non-summer months since I haven’t used less than 1000 kWh in the past two years.  I should check my UPS and see what it’s showing for my daily usage from just the lab.  Although I still haven’t plugged my firewall/switch/modem into it yet (kids don’t like me turning off the internet terribly)….

 

 

Userlevel 7
Badge +8

Lucky guys!
Im setting up my lab with super low power / consumption hardware, 
the electricity bill is killing me here in Spain!!

Does anyone have spare ram for a Proliant Gen 8??
128GB extra would be soooo nice!

🤣🤣🤣

cheers!

🤙

How much is power there? It’s about .09/kWh where i live. in CAD.  so $0.07 USD roughly.

 

Plus we can use the heat in the winter to help heat our houses hahahaha

Userlevel 7
Badge +8

Lucky guys!
Im setting up my lab with super low power / consumption hardware, 
the electricity bill is killing me here in Spain!!

Does anyone have spare ram for a Proliant Gen 8??
128GB extra would be soooo nice!

🤣🤣🤣

cheers!

🤙

Userlevel 7
Badge +6

Nice.   Previously I had a set of FC switches and a small SAN.  The opportunity I have now is infract an SSD SAN. I think I might actualy jump on it the more I think about it. I don’t know that many people that have 200TB at home worth of SSD.  I’ll connect it to my home assistant to monitor power maybe and do a blog post about power consumption and why owning too much storage is less efficient than the cloud or something haha. 

I have an old Equallogic PS6210E that is going to replace the PS6000 in the work lab, but these things are big with a lot of drives and pull a lot of power, and honestly kills the runtime I have on the UPS.  I just racked up a new PowerVault ME5024 for production which will free up a couple of PS6100 arrays, but not really better than the PS6210E except one has a couple of SSD’s in it for a cache.  I also have an old HPE MSA and two Equallogic PS6500’s laying around, but are not of much use and will probably just end up at the recyclers at some point. I am also going to replace the 5 or 6 PowerEdge R610 servers that I have in a cluster with a couple R720’s and possibly an R820.  I actually have four R720’s but really only need two for the lab, and the other two and the R820 can be used for swing servers, crash kits, and for testing a Linux Hardened Repo in the lab.  I’d also like to try out the virtual edition of a Quantum DXi deduplicating appliance on one, but time has prevented me from really giving this much thought.

All of these old arrays (and servers) are pretty power hungry, and while power is relatively cheap, it kills the UPS runtime as well.  SSD’s may not be too bad for power draw and maybe can be more reliable without much for moving parts, so assuming the wear isn’t too bad on those disks, that could be a great array for a lab environment.

Userlevel 7
Badge +8

Nice.   Previously I had a set of FC switches and a small SAN.  The opportunity I have now is infract an SSD SAN. I think I might actualy jump on it the more I think about it. I don’t know that many people that have 200TB at home worth of SSD.  I’ll connect it to my home assistant to monitor power maybe and do a blog post about power consumption and why owning too much storage is less efficient than the cloud or something haha. 

Userlevel 7
Badge +6

Funny how this came up...last weekend I finally got around to installing a 20amp circuit in my garage so I could use a Tripp-Lite 2000VA UPS that I had laying around and got most of my gear plugged into it.  I had planned on sharing, so now seems as good time as any to post some pictures.  The homelab is pretty crude as it sits on top of a refrigerator in my garage and is subjected to extreme hot and extreme cold temperatures and humidity (and lack thereof).  But it gets the job done and really this hardware is solid.  The R610 that this R520 replaced ran for 3 or 4 years after it was retired from my client’s site, so in all, it was finally about 10 or 11 years old when it let out the magic smoke. 

Please don’t mind the disaster that is my garage…..I’ve learned that if you have a lot of space, you keep a lot of crap.

 

Homelab on top of the garage fridge.  Tripp-Lite 2000VA UPS, Dell PowerEdge R520 running VMware ESXi 8.0, Extreme Summit 1Gb POE 48-port switch, Ubiquiti Unified Security Gateway Pro, Synology 2-bay NAS connected via NFS for primary Veeam repository.  The CenturyLink Zyxel DSL Modem/Fiber router is a cold spare in case the Ubiquiti USG fails.  Onkyo receiver on top of everything is used to power some outdoor speakers in the back yard and indoor speakers in the garage and audio signal is supplied by a Google ChromeCast audio.

 

Side-view of the home lab with VMware ESXi 8.0 console on the monitor.

 

Rear view of the homelab equipment with a Geist PDU and and the newly installed 20amp circuit on the ceiling to power the UPS.

 

Userlevel 7
Badge +6

Cheap home lab approved, (no cost) but the power bill and fan noise may not be worth it.  

 

One things I hate about HPE (among a few things) is that if you put a non-HPE branded/approved component, all of the fans will run full out on the server.  This included NIC’s and I think memory...not sure about disks.  If you want to keep it quiet, you need to stick with HPE components.

 

I think i’ll keep a second as a spare powered down. I also have the opportunity to grab a pretty large SAN, but I’ll most likely decline as the power draw is really overkill for my homelab. 

 

This is a good plan.  I have a spare host as well since I don’t have a need - though I’m toying with the idea of loading the Linux Hardened Repo on an old Dell R610 I have with a crappy RAID controller that has no cache memory.  But it seems like overkill for home since I have a copy of my data going to Wasabi which is immutable.

 

I’ll keep the NUC for home assistant, cameras, and things that run 24/7. I think the second ESXI lab will only get powered on for test purposes to keep costs down.    With my work lab being so powerful, i find my homelab getting used less and less these days. 

This is what I have going on as well….my work lab has a lot of stuff going on, and is ready to be refreshed when I can find the time, but I don’t need much running at home.

Userlevel 7
Badge +8

I just got the opportunity to grab several HP Gen9’s with about 512GB of memory each.

Cheap home lab approved, (no cost) but the power bill and fan noise may not be worth it.  They run between 170-230 watts average, so 500watts will average about $40 a month where I live.

 

I think i’ll keep a second as a spare powered down. I also have the opportunity to grab a pretty large SAN, but I’ll most likely decline as the power draw is really overkill for my homelab. 

 

I’ll keep the NUC for home assistant, cameras, and things that run 24/7. I think the second ESXI lab will only get powered on for test purposes to keep costs down.    With my work lab being so powerful, i find my homelab getting used less and less these days. 

Userlevel 7
Badge +8

For avoiding networking issues, I normally virtualize a pfsense or OPNsense and NAT the “external” network, and then I start deploying host only networks or NAT networks, keeping them separated from the Bridge network, so I don't get any dependency on the local “real”nics and the IP ranges.

Userlevel 7
Badge +9

In addition to what Paul has said, remember private IPs’ are not routable. To get this to work correctly, you must have the right settings in place with any of your type 2 virtualisation solutions.

Userlevel 7
Badge +20

Hi Everyone,

It's me again. i am just wondering why everytime i change the IP Address I am not able to access the web Url using that IP address for IPv4 and when i set it to static.

 

I used Bridge as connection. However, after resetting it back to original, i am able to access it.

Hi @seanrockvz13, can you give us some more info on your setup please.

 

You mentioned you’ll use VMware Workstation on Windows. As this is a laptop, it’s quite possible that you’ve got multiple NICs. Bridged mode is, by default, provisioned in auto-bridge mode, so if you’ve got multiple network interfaces such as WiFi & Ethernet, could it be using the wrong interface? If you have a specific interface you want the traffic to traverse via, you should configure the bridge to reflect this.

 

Am I correct in assuming your issues with IP addressing are related to your ESXi VM within VMware Workstation, or is it a layer deeper and virtual machines running inside your ESXi VM?

Userlevel 2
Badge

Hi Everyone,

It's me again. i am just wondering why everytime i change the IP Address I am not able to access the web Url using that IP address for IPv4 and when i set it to static.

 

I used Bridge as connection. However, after resetting it back to original, i am able to access it.

Userlevel 7
Badge +9

Good points! You may have to enable virtualization in the BIOS if it hasn’t already been enabled. I would recommend VMware Workstation and not the player or VirtualBox! This will help should in case you ever run into this issue: https://techdirectarchive.com/2022/02/07/enable-virtualization-in-bios-determine-if-the-intel-vt-x-or-amd-v-virtualization-technology-is-enabled-in-bios/

 

Yes, im going ahead with VMWare Workstation 16. :) Thank you for the link! I was able to install and set it up. Does that mean i can skip this step?

You will be able to determine this when you get an error 😁 

Comment