In this episode, Chris chats with Abel Gordon, Chief System Architect at Lightbits Labs, discussing the challenges and benefits of building a virtual storage area network (SAN) on public cloud infrastructure. Lightbits originally developed the NVMe/TCP protocol and uses this feature to build virtual SANs using public cloud instances. This is a topic we first looked at in episode #210, so it’s good to get a practitioner’s experience.
Modern public cloud now features fast networking, low-latency NVMe and high-performance virtual and physical instances. Unfortunately, NVMe devices are ephemeral and any provisioned storage in the cloud is charged at full capacity. For users of on-premises SANs, the lack of thin provisioning may be an unwelcome surprise.
Why build a virtual SAN, other than to save storage costs? There’s a lot more involved, including delivering resiliency, scalability, targeted performance and capacity. Abel discusses the benefits, then goes on to enumerate the challenges involved when building on vendor-owned infrastructure. Finally, the discussion moves on to how Lightbits’ software is deployed and operated, including the managed application capability in Microsoft Azure.
For more information on Lightbits Labs, visit the company website at https://www.lightbitslabs.com/
As Abel, suggests you can contact him on LinkedIn or email him directly at [email protected].
Elapsed Time: 00:51:15
Timeline
- 00:00:00 – Intros
- 00:02:07 – Why build a virtual SAN in the public cloud
- 00:04:30 – SANs balance out and fully exploit available performance resources
- 00:06:36 – Public cloud charges for performance and capacity
- 00:08:12 – On-premises SANs offered full flexibility to manage all metrics
- 00:09:25 – Cloud autoscaling combined with software gives much more flexible storage
- 00:13:27 – The on-demand nature of cloud works well for scaling SANs
- 00:14:40 – New cloud features – NVMe, fast networking and NVMe/TCP have enabled solutions
- 00:17:19 – What is NVMe/TCP?
- 00:20:50 – What challenges are there in delivering a SAN on public cloud instances?
- 00:24:03 – Cloud providers optimise for their system, not for your application
- 00:25:02 – What operating system issues exist when building a virtual SAN?
- 00:28:27 – Userspace operation requires a different programming strategy
- 00:31:00 – NUMA awareness is essential, even in the public cloud
- 00:32:46 – Each new instance type requires retesting and validation
- 00:35:53 – What is the Lightbits solution and how is it deployed?
- 00:37:00 – NVMe cloud drives are ephemeral
- 00:41:35 – Snapshots work differently in the public cloud
- 00:44:12 – Is Lightbits dedicated or HCI?
- 00:46:32 – How is the solution consumed?
- 00:47:29 – Azure offers management application capability
- 00:50:36 – Wrap up
Related Podcasts & Blogs
- Is the Public Cloud Becoming More Reliable?
- Zesty Optimises AWS EC2 EBS Storage
- Storage QoS In The Cloud
- #97 – Building Storage Using NVMe/TCP with Kam Eshghi from Lightbits Labs
- #121 – NVMe 1.4 Deep Dive Part II with J Metz
- #210 – Building SANs in the Cloud
Copyright (c) 2016-2023 Unpacked Network. No reproduction or re-use without permission. Podcast episode #cv54.
Podcast: Play in new window | Download
Podcast (storageunpacked): Play in new window | Download