Published On - September 21, 2023
Building Open Source Software: Web3j, an Origin Story
This month marks a big milestone for me in the Ethereum community. In September 2016, I announced the open-source library, Web3j, that I had authored to the world. Web3j was created out of a clear need that I saw, but the outcome of creating this software was nothing like I expected.
In December 2015, I left the corporate world of banking. I'd spent the previous 10 or so years building technology platforms in the financial services industry, a place that was full of smart people, but a working environment that wasn't conducive to my way of being — too many meetings and processes.
I'm the kind of person who thrives with variety and needed to find an out (in fact in the years since then I’ve being formally diagnosed as being on the autistic spectrum, which no doubt plays into this).
The first out was a failed startup, Huffle with a couple of great co-founders. We tried to launch a new home loan product in Australia, but quickly learned that convincing a bank to launch a product based on a financial model you have is not easy. Especially when a) you’re a tiny startup and b) this product won’t significantly move the needle initially for them as far as volume is concerned.
As a result of this failed experiment, I'd decided to start looking into this blockchain thing as Huffle was wound up.
What I saw then (this was around May time 2016) was that Ethereum seemed to be the blockchain platform that most of the buzz was about — if you searched blockchain in Google, I'm pretty sure Ethereum was the first thing that came up. Although it wasn't particularly clear what it was from the website back then.
My Ether mining rig
Anyway, I started digging around, built a cryptocurrency mining rig (which I can talk about some other time), and then started getting more involved in my local community — the Sydney Ethereum Community.
In all those years working in banks, I'd been building on the Java platform. Since the early 2000s, Java has been adopted very heavily in enterprises due to its ease of deployment and automated memory management compared with previous programming languages.
This meant that by the 2010s, significant parts of our modern financial infrastructure ran on the Java platform. It was further boosted by Google using it as the basis of their Android mobile platform.
Back when I was continuing my foray into blockchain and Ethereum, there was a lot of talk about how blockchain was going to be a transformative force for financial services.
In all those years working in banks, I'd been building on the Java platform. Since the early 2000s, Java has been adopted very heavily in enterprises due to its ease of deployment and automated memory management compared with previous programming languages.
This meant that by the 2010s, significant parts of our modern financial infrastructure ran on the Java platform. It was further boosted by Google using it as the basis of their Android mobile platform.
Back when I was continuing my foray into blockchain and Ethereum, there was a lot of talk about how blockchain was going to be a transformative force for financial services.
Articles like this were common, even back in 2016
Given my experience, I noticed a clear gap in the market. How were all of these financial services companies which had spent the previous 15 years building platforms on Java going to hook them into the Ethereum network — in my view the most promising blockchain technology?
Scratching beneath the surface, it was clear there was no easy way for Java developers to work with Ethereum. I was shocked! It was such an obvious opportunity! Surely someone else must have thought of it by now? After all, this was midway through 2016, the Ethereum network had been live for a year (it launched on the 30th July 2015).
I spent time researching numerous code repositories on GitHub and found libraries for JavaScript, Microsoft's .NET platform, but nothing for Java (Note: there was an early Ethereum node implementation — EthereumJ, but this was for running a node, not simply integrating with the network which is different).
I was jittery with excitement. Usually, the problem that engineers face when trying to do greenfield development (building something from scratch), is that invariably someone has beaten you to it and it doesn't make sense to handcraft everything yourself.
It was time to start creating a prototype.
In order to talk to a node on the Ethereum network, you have to connect with a protocol called JSON-RPC. JSON-RPC is the same protocol that is used to communicate with nodes on the Bitcoin network (a number of implementation details of Ethereum were copied from Bitcoin, such as JSON-RPC, and the Secp256k1 elliptic curve used for cryptographic operations).
There were a ton of different method requests and responses I needed to implement, ranging from getting information about the node software being run to the transactions within a block. Once I'd implemented a few and still hadn't found a library doing anything similar, I was ready to commit, and make this project a reality.
I spoke to my wife, we'd had our first kid a few months earlier and I said to her that I'm going to need to properly get my head into it for a few weeks to get this first version done. This meant spending as much time as was humanly possible devoted to the library.
I was at home, so when I needed a break I could hang out with my son which was fortunate in these early years of his development.
In those weeks the days just whizzed by, I'd get in front of my computer as early as I could, put my cans on, get into the zone, and before I knew it another day was over.
The first commits to Web3j
This carried on until I had something that provided the core functionality to work with Ethereum. In my naïvety, I had thought the JSON-RPC implementation was all that was required.
However, I quickly discovered that to do anything meaningful with Ethereum, you had to also support the creation of wallets, signing of transactions and the ability to create, deploy and interact with smart contracts. This kept me busy. Especially as I didn't just want to create any old integration library, I wanted to create a high-quality library, with excellent documentation and build a community around it.
The naming of the library was easy — Web3j. The reason is that almost anyone who worked with Java was aware of the major logging library — Log4j, plus you also had the JavaScript integration library web3.js. Web3j spoke to what the library was perfectly in my view.
Web3j was formally announced to the world via the Ethereum subreddit on Sunday 18th September 11:26:35.
Web3j’s first ever issue!
Momentum slowly started to build. By the end of November, four different people I had never met before had contributed to the library, and I had to start focusing my energies beyond just writing the library and fixing bugs.
The first pull requests submitted by people I had never met to Web3j!
It was time to invest time in the documentation and think about how I could promote Web3j and build a community around it, such as the below talk I gave in early 2017 (which has has over 60k views).
This is where the hard work truly started, and 7 years in I’m happy to say the project is still going strong. We’ve been fortunate that almost 200 people have made contributions to the library and it’s been downloaded over 2 million times! Whilst I’m no longer contributing to it myself these days, the team at Web3 Labs are. There is also a steady stream of new users.
Earlier this year I was at an event where staff from a major American bank mentioned how useful they were finding Web3j for their work with blockchain. This and numerous similar conversations over the years reaffirm that I made the right decision all those years ago in creating it.
The vast majority of open source software is a gift to the world. It solves a problem that people have. In Web3j’s case it was using one of the world’s most widely used programming languages, Java to develop on Ethereum. I’m proud that I took the time to create this all those years ago, and I wouldn’t change anything in that respect.
