Home
Creators
Search Recent Random
Posts
Search DMs Popular Hash Lookup Tags Random
Importer
Import FAQ
Register Favorites Keys Review DMs Logout
Community
Telegram ThePornDude
Home Creators Posts Import Register Favorites Logout

160TB Server with Linus Tech Tips! - Smarter Every Day 222 (Patreon)

Published:
2019-07-30 20:00:04
Imported:
2026-06

Content

Linus finally came by and helped me get my data storage game sorted out.  It was pretty abysmal, so this is a huge development.  

I didn't mention this in the video, but I ultimately decided not to put the old data on this server.  My thought is that if I only put NEW stuff on this server, I'm more likely to be safe in the future for a longer amount of time. 

Another thing I'd like to point out is that your support on Patreon makes videos like this possible.   Thanks for letting me do stuff like this.  I try not to be clickbaity etc with the videos so this is a pretty level shot.  Thanks for freeing me from that kind of thinking.

~~~~~~~~~~~~~~~~~~~~~

It finally happened....  If you also wear sandals over socks, you can subscribe by clicking this link: http://bit.ly/Subscribe2SED

Look Below for more links!  ⇊

Original Tweet:

https://twitter.com/smartereveryday/status/1043578289288355841

Thanks to:

45 Drives - Customizable, economical, high density storage servers

https://www.45drives.com/

unRAID - Safe, protected data storage, based on Linux, configurable

https://unraid.net/

Seagate - 5 year warranty, vibration-resistant, data-recovery warranty

https://www.seagate.com/

Linus's video when he visited the US Space and Rocket Center

https://www.youtube.com/watch?v=olRF5Ckaga0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

GET SMARTER SECTION

Standard Raid Levels:

https://en.wikipedia.org/wiki/Standard_RAID_levels

How Unraid works:

https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Unraid

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Tweet Ideas to me at:

http://twitter.com/smartereveryday

Smarter Every Day on Facebook

https://www.facebook.com/SmarterEveryDay

Smarter Every Day on Patreon

http://www.patreon.com/smartereveryday

Smarter Every Day On Instagram

http://www.instagram.com/smartereveryday

Smarter Every Day SubReddit

http://www.reddit.com/r/smartereveryday

Ambiance, audio and musicy things by: Gordon McGladdery 

https://www.ashellinthepit.com/

http://ashellinthepit.bandcamp.com/

If you feel like this video was worth your time and added value to your life, please SHARE THE VIDEO!

If you REALLY liked it, feel free to pitch a few dollars Smarter Every Day by becoming a Patron.

http://www.patreon.com/smartereveryday

Warm Regards,

Destin

Files

160TB Server with Linus Tech Tips! - Smarter Every Day 222

It finally happened.... If you also wear sandals over socks, you can subscribe by clicking this link: http://bit.ly/Subscribe2SED Look Below for more links! ⇊ Original Tweet: https://twitter.com/smartereveryday/status/1043578289288355841 Thanks to: 45 Drives - Customizable, economical, high density storage servers https://www.45drives.com/ unRAID - Safe, protected data storage, based on Linux, configurable https://unraid.net/ Seagate - 5 year warranty, vibration-resistant, data-recovery warranty https://www.seagate.com/ Linus's video when he visited the US Space and Rocket Center https://www.youtube.com/watch?v=olRF5Ckaga0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ GET SMARTER SECTION Standard Raid Levels: https://en.wikipedia.org/wiki/Standard_RAID_levels How Unraid works: https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Unraid ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tweet Ideas to me at: http://twitter.com/smartereveryday Smarter Every Day on Facebook https://www.facebook.com/SmarterEveryDay Smarter Every Day on Patreon http://www.patreon.com/smartereveryday Smarter Every Day On Instagram http://www.instagram.com/smartereveryday Smarter Every Day SubReddit http://www.reddit.com/r/smartereveryday Ambiance, audio and musicy things by: Gordon McGladdery https://www.ashellinthepit.com/ http://ashellinthepit.bandcamp.com/ If you feel like this video was worth your time and added value to your life, please SHARE THE VIDEO! If you REALLY liked it, feel free to pitch a few dollars Smarter Every Day by becoming a Patron. http://www.patreon.com/smartereveryday Warm Regards, Destin

Comments

Nick Vahalik

I'm waiting for Linus to drop something...

Kyle Sisk

Dude, we have a couple of these at DCMP and they're amazing. It took a bit to tune the NICs in our editing machines to take full advantage of the10Gbps network, but the effort was really worth it.

Nick Vahalik

But seriously, this is nifty! Sounds like you got a great setup and it should make your life a LOT less stressful.

Nick Vahalik

Hey Destin: totally random question, but that poster on the wall behind you... is that one of those massive electromagnetic spectrum posters?

Tyler Boespflug

I viewed the original tweet before watching the video. Not going to lie... The pictures made me gasp a bit. Glad you got your data organization handled.

Adam D'Alessandro

oh man, the old system would give me so much anxiety. I hope you expand so you can add the old data onto the server like the original plan.

Maarten Daalder

I was wondering what happend to the Apollo collab between the two of you.

Rich Jeanes

Yeah... The problem with your scale is that you can't expand that storage server any more with all of those drive bays full. If you want more storage there, you just need a bigger/another server.

Ben V

Agreed with Andrew! Make sure you've got off-site backup, as I've always been told... and learned the hard way... raid != Backups

Nick K

3-2-1 rule! At least 3 copies of your data, on at least 2 different types of media, and at least 1 copy in a different (offsite) location. In addition to drive failures impacting your RAID array you can experience other hardware failures that can impact the integrity and availability of your data, and having it directly accessible to your computers locally leaves your data vulnerable to ransomware.

Dan & Jen Baizel

I realized your phantom generated a ton of data, but I never really thought about all of the old footage until now. By the way, as someone who abuses Excel sheets for everything I approve of your catalogue system.

Sean Chittenden

I love the channel, but this video left me concerned. As someone involved in storage, for this amount of capacity, three things come to mind: 1) Unraid uses ReiserFS. How does this system handle "bit-rot"? What interval is the data being scrubbed to ensure the integrity of the bits? 2) Is there something on top of ReiserFS doing erasure-coded storage? 3) Depending on the quality of your power, I'd plug that server into a small UPS to do power line-conditioning in order to prevent brownouts or short-strokes of the HDD drive-heads. I know when I use ZFS at home on my storage box, I was running into dozens of correctable storage errors per week until I put a UPS in and the number of reported errors from a zfs scrub went to zero. Regardless, for 160TB, I'd look into something like ZFS or something erasure-coded based like minio that can do scrubbing and self-healing of corrupted blocks. Good luck!!!

Invisible Jazz

You guys together in a video? This is madness!

smartereveryday

Yes backblaze was something I investigated but it's cost prohibitive.... and I need to have something local anyways to backup. What's your backup solution for your 18TB?

Sean Chittenden

Doh! Sorry about that. Here's a decent pop-press intro to bit rot: https://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/ The TL;DR: is that after roughly 10^14 bytes read can cause an uncorrectable read error because the underlying bits of a hard drive flip for one reason or another. That's only once every ~12TB. The actual name of the stat is called the "UBER", or Uncorrectable Bit Error Rate (a rather difficult term to search for in this day and age). The other interesting metric is the MTBF, or Mean Time Before Failure. But, this is a probability game and there needs to be some active correction to prevent entropy from causing data loss. I've managed petabytes of storage for $JOB and see this in data centers as well as in residential situations. Uncorrectable drive errors are a foe and constant threat. This whole area of computers is a descent into madness because at this scale of density, physics is not your friend unless you start to embrace statistics. With 160TB of data, these probable failure rates are something you will almost certainly have to contend with. https://wccftech.com/backblaze-releases-hard-drive-findings-for-q1-2019-hard-drive-failure-rates-for-all-major-suppliers/ https://www.computerworld.com/article/2846009/the-5-smart-stats-that-actually-predict-hard-drive-failure.html Which is all just a long way of saying, you probably want to check to make sure that your unraid install is using ZFS under the hood. Even 45 Drives recommends the use of ZFS or some way of detecting and remediating this all-but-guaranteed effect: https://45drives.blogspot.com/2019/04/hardware-raid-vs-software-raid.html

Nick Vahalik

You kinda have to have the same basic approach as dealing with security. What is your threat model? What are you worried about? Are you worried about bit-rot? Fire? Theft? Are you worried about a hacker encrypting your data? Your threat model will have a lot to say about how you approach those things. I've been watching Linus for about 6 months and he's pragmatic and IMO he has a good trade-off between solutions that are good and also cost-effective. He did a video about the Jellyfish a couple of days ago that I think underlines what he does and why he does it. Something like Synology or even Drobo makes high-redundancy really accessible and the Stornado + the Enterprise drives are a great option. They will protected from what Destin's most likely issues are: bit-rot and misplaced drives.

Nick Vahalik

Destin, you'll probably get different opinions, but here are a couple of options for data at your scale: 1. Backup to M-Disc Blu-Ray or something similar. This would allow you to offload 25-50 GB per disc. M-Disc are rated for archival purposes with a shelf life if 50+ years. 2. Use tapes. LTO-7 & 8 give you multiple TB (depending on how compressible the data is) per tape. Offload as many tapes as you need for archival footage and just store those in a cool, dry place. (IMO, this is probably the best option from a cost and usability perspective) 3. Duplicate hardware offsite - probably not what you'd need but at a previous job we essentially did this. Grabbed a duplicate of our hardware, stuck it in a datacenter, and then just mirrored the data over to it. You'd need fast internet but it is effective and basically just requires that you scale both at the same time.