The Netflix cloud team loves OSS — and would love to stop building it
Netflix is in known in some (albeit geeky) circles as much for its advanced Amazon Web Services architecture and open source software as for its streaming video service. But some members of cloud team would love the company to stop building its own tools and start using commercially available services from AWS.
Ruslan Meshenberg and Andrew Spkyer, two senior members of that team, came on the Structure Show podcast this week to talk all about open source at Netflix. Essentially, they said they’d really like to see AWS create more tools that live up to Netflix’s needs, but in the meantime will keep on building new open source tools and trying to make them easy for other companies to use. Here are the highlights.
Netflix open source, all packaged up as Docker containers
The newest piece of open source tech out of Netflix is a project called ZeroToDocker, which is intended to simplify the process of deploying and understanding the company’s other various tools. It’s not meant to power production services, but just to let someone spin up, say, cloud-management framework Asgard on a laptop and learn it in an hour instead of over weeks.
“The Netflix technologies are designed to be run at very large scale … but running a simpler deployment and actually understanding it as a new user to the open source is pretty tough,” Spyker said. “The breadth of the 40-some, maybe 50-some projects is a lot to get your head around.” (The whole Netflix open source library is available on the NetflixOSS GitHub repository.)
Still, he added, just over a week after announcing ZeroToDocker, downloads are ramping up and at least a couple companies are putting it to the test. A cloud-management company called Servmon has incorporated the container images into its software for customers to use, and “some folks at Microsoft have already gotten this up and demonstrated it on Azure with CoreOS as the base operating system.”
Andrew Spyker
Doing open source right
Netflix, like Facebook (whose open source lead James Pearce was on the Structure Show a couple months ago), is all about doing open source right rather than just technically doing open source. Responding to a question about Microsoft’s recent decision to open source its .NET programming language, Spyker said, “There’s open sourcing and just sort of throwing it over the wall, and there’s open sourcing it of doing it in a community. Putting it out on GitHub and trying to build a community around it, I think that’s almost more important than just the pure open sourcing part.”
Meshenberg added to that sentiment, discussing how Netflix approaches the topic:
“What you have to realize is building a community takes an order of magnitude longer and more effort than putting the technologies out there. … At this point, let’s say one of the indications of the size [of our community] is our meetup community, which is over 1,500 members and growing. [I]t has reached, I would say critical mass so that every single open source meetup that we host is absolutely a full house. …
“But that takes engagement from all sides, it’s not that we just throw things over the fence and say they’re open source now. We do have to provide a certain level of support and internal engagement to make sure we actually use that very same open source code as much as we want external people to do.”
Ruslan Meshenberg
But we’d rather not do it at all (yeah, right)
However, Meshenberg and Spyker, like many other internet architects and engineers, claim they’d rather not have to build so many tools and just use what’s already available from AWS. (Although some only seem to pick up the pace of building new stuff.)
“As time progresses, and they grow and their scale gets bigger, hopefully we’re becoming a smaller and smaller fish in the pond. That’s our goal, as well,” Meshenberg said. “We don’t want to be the unicorn that we’ve been for the last few years, we’d just like to use whatever common solutions that exist.”
A small sample of what’s available from Netflix.
Talking about database technology — specifically the shortcomings of AWS’s original SimpleDB service – he specified an instance where Netflix might have done something different: “Had [Amazon DynamoDB] existed at the time we needed to store data at scale, perhaps we would not have gone with Cassandra. … We are constantly evaluating things that if we don’t have to do, I’d rather not do. And just because we’ve never done it that way, it’s never a good enough excuse.”
But the fact that Netflix uses open sources technologies such as Cassandra, which wasn’t actually built at Netflix, suggests maybe the company is serious about not building so much stuff. Assuming, of course — and perhaps unrealistically — that any commercial tools are good enough for the job.
“There’s a certain level of scale, availability, operational visibility that goes into any deployment at Netflix, and those have to be there at the level of scale and complexity that Netflix is at,” Spkyer said. “And once they’re there, then building it ourselves just doesn’t make anymore sense.”
Image copyright Netflix.
Related research and analysis from Gigaom Research:
Subscriber content. Sign up for a free trial.
How the mega data center is changing the hardware and data center markets
How to resolve cloud migration challenges in physical and virtual applications
Sponsored Research: How direct-access solutions can speed up cloud adoption
More From paidContent.org
