Commit 1 to 1000 and beyond: Two years of maintaining an open-source project

Back in June 2020, I started writing what is now known as Skytable, a NoSQL database project. Ever since, I have been maintaining Skytable, mostly in my free time and have recently been spending a lot of time on it. Here’s a little story on my two years of experience in maintaining an open-source project: what it’s like, the highs and lows and the future.

Before I go deep into things, what is open-source? The definition with regard to the license used today would be “any project that uses an OSI-approved license.” However, without introducing indirection with a license, open-source can be generally looked at as “any software that is licensed in a way such that the creator of the work grants to other users the rights to study, modify and distribute their code.” Of course, the license used by the project will greatly dictate how that code can be used, but that’s the general idea.

The beginnings

Mid 2020 to Early 2021

Contemplations

But well, why does someone start or contribute to an open-source project in the first place? Well, there are many reasons and I have noted a few here:

1. They want to get into the vibrant world of open-source: Thanks to the convenience offered by GitHub for hosting open-source projects, building and sharing a project has never been easier, unlike setting up mailman and then asking people to contribute over a mailing list. Don’t get me wrong, a mailing list is obviously very effective! The Linux Kernel and projects surrounding Linux in general all use mailing lists for development and discussion. However, for newcomers it may be a hustle at times
2. They solved one of their own problems: This is another very plausible take. Say you were writing a random number generation library, and you asked yourself, “Oh, this can be useful for others too.” Following that, you push your code and make it available for anyone to use.
3. They’re out for a job! And hey, that isn’t a bad thing. Building and contributing to open-source projects can be invaluable, provided that they make useful contributions. Non-useful contributions include the people who spammed open-source developers with bogus PRs for winning t-shirts¹ (and yes, we got a couple too)
4. They’re out to solve a problem: This is like #2 except that the problem isn’t solved. This individual is out to solve a problem, with help from the community.

For me, it was #4. I have had multiple complaints with existing databases, primarily surrounding the data model, performance and scalability, and being a person who studies databases, I decided that I should write one (actually the second one; I have previously worked on similar projects). PS: I also have a more philosophical reason for being a part of open-source which I leave the reader to find somewhere in this blog.

Learning: First find the problem, then look for a solution!

The source

Most people don’t know, but Skytable wasn’t originally called Skytable. Instead, it was called TerrabaseDB. If I remember correctly, the first commit was maybe one of the funniest pieces of code I’ve ever written: it implemented a REPL that read in lines from stdin, categorized the action kind and arranged into an appropriate structure. It did just that. Ofcourse, soon enough I wrote the first version of the Terrapipe protocol (which used an infamous metaline that I later screamed about) and the server that could actually run the actions.

Learning: There’s no harm in starting small!

The community and the starry adventure

In the first few months, there was little to none “significant growth.” And, how do you judge growth for an open-source project? Thanks to GitHub’s prominence in this era of open-source, welcome, meet GitHub Stars. It’s like Facebook’s likes for developers. Every time someone stars your project, you get that burst of dopamine. GitHub stars have kind of become like a seal of approval for open-source projects — the more stars you have, the more popular and “legit” your project is². But for an open-source project maintainer, what does it mean? I’m not sure about others, but for me stars are a form of encouragement and a feeling of appreciation by the community. Because afterall, when you’ve set out on building an open-source project, there’s nothing more you’re out for other than people appreciating what you’ve built.

And, so following this line of thought — I stood around for more stars, but the wait wasn’t entirely successful; atleast not back then. But I kept going because it was fun! Surely, having more people following the project would’ve encouraged me, but the problem in itself was fun to solve.

Learning: Stay at it!

Uphill, downhill and the middle grounds

Early 2021 to Today

The source, again

After many iterations, the protocol was finally much better than what it used to be. It still had length headers, but it was far less funky than it used to be. And we finally had persistence. I also added several other utilities like background saving, snapshots, configuration and more. Also, I took the time to set up a documentation site (it was then built with mkdocs) to make it easier for folks to set up things on their machines. We eventually switched to Docusaurus and it has been a pleasant experience, thanks to its easy versioning. TerrabaseDB switched names to Skytable, especially inspired by Linux landing on Mars.

A few more months down the line, I finished writing the next protocol Skyhash and we released keyspaces, tables, volatile tables, authentication and a much better persistence system.

Also, one thing to note: while several folks tend to compare Skytable to key/value stores, Skytable’s goal has never been to be just a K/V store. There’s much more coming.

The community, and the criticism

Firstly, if you’re going to share something on the internet, it’s going to attract criticism and by sharing it on the internet, you’re voluntarily putting yourself into that chaos. Undoubtedly, this post itself will attract some criticism.

But guess what — criticism is great, but only when it’s constructive. But then there are some aimless criticisms like:

• “Oh so you built a map structure with network access,” to which I told myself, yeah, MySQL is definitely just a BTree with network access. (yes, a key/value store is relatively simpler but definitely not a map with network access 🤡)
• “You don’t even have clustering in 2022.” One should realize that: (1) clustering is a non-trivial feature (2) I’m doing this in my free-time

There are better ways to ask for it. And when you can’t appreciate what someone has created in their free time, atleast do them a favor by not making such futile comments. Wouldn’t it have been better if someone said “You should definitely work on supporting clustering ahead.” The moment I see such a message, that thing almost always makes it to my roadmap. The same happened for PUB/SUB which is coming soon upon request by several folks from our Discord community. This traverses me to the next node in my line of thought.

Running an open-source project is no less than running a company

I often tend to think that running an open-source project, while attempting to spread the word is no less than running a company. How do I justify that? Here’s how:

1. You’re the Sysadmin: You set up your website, put in DNS records, fire up a VM to host a blog like Ghost, or even setup a self-hosted short-links tool
2. You’re the DevOps Engineer: You set up CI/CD, auto upload of assets when pushing to tags and maybe even run a CI agent on your VM
3. You’re the Chief Marketing Officer: But ofcourse! You’ll be sharing updates on social media, building banners and other assets for “branding”
4. And obviously, you’re the software engineer: I’m sure that doesn’t need any further expansion

Like you can see, it’s almost like running your own startup! A full-time responsibility, except that no one pays you for it.

For one, I think open-source project maintainers can become great entrepreneurs. We wear multiple hats and take on multiple jobs. We live with the criticism.

Feature requests, and support

Firstly, don’t me wrong — I love feature requests because that way I can know what the community wants. And when you’re building for people (aren’t all of us?), you need to know what they want. My favorite quote here is:

And, one of the things I’ve always found is that you’ve got to start with the customer experience and work backwards for the technology. You can’t start with the technology and try to figure out where you’re going to try to sell it. […] — Steve Jobs

However, some feature requests are particularly off-putting, especially when you’re doing something voluntarily in your free-time. Here are some of the ridiculous ones (either from some chat room, e-mail or GitHub):

• Support clustering immediately
• I really need transactions. I want them right away
• Support feature X from Database Y, ASAP
• Please support MySQL (and, I totally lost it on this one)

These wordings are a little annoying. Especially the impassive deadline that I never signed up for. However, here’s one thing that I’d like to tell all people who’re getting to grips with open-source projects: sometimes, the people who use such wordings didn’t intend the rush in the first place; it is important to note that not all developers speak English as their first language and often use translators or other tools that lead to such statements. Whenever I do encounter such situations, I always respond responsibly. We’re all human, so let’s think others are human too! But when things do get out of hand, don’t hesitate to say “No” or “We’ll look into this later.”

My takeaway

One question I’d like to answer is what I gained from maintaining the project. For me, the most important thing has been community. I have talked to and met a lot of great people, and several fellow maintainers too. I have also made some great friends along the way. Imagine getting to know someone on the other side of the world without a dating app 😂! (I still remember a joke where someone apparently proposed their partner with a pull request). Truly, open-source is something can bring us together, despite our differences. I also worked on security issues that we discovered with some fellow community driver maintainers. Maintaining an open-source project has taught me a lot, and I can’t wait to see what I learn and build in the many more years of open-source ahead!

You can find Skytable on GitHub, what we’re working on next and me on Twitter.