Technology Evolution @ Zenatix (Software Stack @ Zenatix, Part-1)

Zenatix recently completed 7 years, starting the journey in December 2013.

On this occasion, I am starting a multi-part series where I will be discussing how the technology has evolved over these 7 years of the existence of Zenatix. I will be discussing this evolution in 3 broad verticals — Software, Hardware, and Data Analytics. The intent of this series is multi-fold:

1. Demonstrate a path on scaling up technology with lessons that apply beyond the IoT vertical.
2. Bring forth the key technology differentiators that we have built in the IoT domain, and open up a healthy debate on how others perceive these technology strengths.
3. To give an inside perspective of our technology to the potential folks who may consider joining us in our journey for a sustainable future.

First up, about the team —

Over these 7 years, the technology team has slowly scaled up in terms of the number of people. Starting with just 1 full-time person and 2 interns in 2013, we kept adding a few people every year and scaled to approx. 10 people in early 2017. Over the years, we have hired a total of approx. 35 people in technology. The technology team size at any point of time has varied between 10–20, where we kept adding a few people every year and some people decided to move on. We have kept the technology team lean and kept our bar high when we recruit — the intent is to put together a team of smart people who can take things to scale without adding a significant headcount. That way the learning for an individual team member is also high.

In terms of experience levels, over the years, there has been a strategic shift. Most of the people who joined in the initial 3–4 years had Zenatix as their first job or were still in the first year since graduation when they joined us. We then started hiring people who had 1–3 years of experience outside of Zenatix to get an outside perspective on how we are dealing with the scale internally. 2020 has been a year where we have hired multiple people with 10+ years of experience both on the hardware and software side. Beyond bringing in loads of tech experience, we believe with these people around, junior people will get a better mentorship experience and as a team, we get richness in perspective on how we should build the processes as we build for further scale.

If this sounds exciting to you and you would like to contribute to the highly scalable IoT stack at Zenatix, while enhancing your exposure to the state of the art technologies, feel free to reach out to us. We are always looking for rockstars looking to make a difference towards a sustainable future. Check out our hiring page for the open roles.

Now coming to technology evolution and picking up Software as the first vertical.

When we started Zenatix, the likes of AWS-IoT and Azure-IoT did not exist.

Before starting Zenatix, I had exposure to a few research projects in the systems area that dealt with building energy consumption (one of the use cases of IoT that Zenatix intends to work on). As part of my research work at IIIT Delhi, we evaluated the architectural designs of these systems from an application perspective and had also played around with them with the small-scale, on-campus IoT deployments. With this knowledge, we picked up one such system, built at the University of California, Berkeley (sMap), and started working upon it. The system was built keeping automation in buildings at the core, which was one of the primary use cases (and eventually became the only use case for us) that we started with. It was architected to talk to different devices (found in building automation space) over the standard protocols and came with an open-source time-series database (with a query layer adapted for the building automation context). All these things helped give us a reasonable context, to begin with on which we built further to create an end-to-end IoT architecture, over time, addressing the business requirements as they kept evolving. Decisions have been taken to add new services or modify the architecture so that the stack can scale up ahead of the business scale-up. Today, we process 200+ million data points on daily basis and we believe the current architecture is capable to scale to much larger data input.

This notion of starting with an existing system in open source and building upon it has served us well through the journey of Zenatix.

It is now part of our culture that for any new service that we need (or an older service that we plan to significantly change to address the complexity that has been added over the years) we do our research of open source options out there, narrow them down to 2–3 options with which we then do some extensive PoC, and then pick one of them (with the right reasons) to extend further.

Let us now go deeper into the initial architecture and the decision-making that went behind parts of this architecture that we need to improve.

Figure 1 — Initial Architecture

Over the initial few years, as the business side was identifying the product-market fit, the objective for technology was two-fold, quickly able to support new use cases from new customers and keep building upon the business logic that will generate value over time.

The initial stack (as shown in Figure 1 above) was a monolith and divided into the edge side and the cloud side. On the edge side, we were clear that we would put in a processor-based architecture (rather than micro-controller-based ones) since that will allow us to remotely add new use cases for the deployments as well as allow us to work with higher level, multi-threaded application. This has served us well over the years. Even today, our standard application consumes less than 10% of the edge resources, leaving a lot of room to add more and more edge intelligence and roll it out at all the 2500+ locations where the hardware is already deployed without having to visit these sites that are spread across 200+ cities in India.

Edge IoT architecture consisted of drivers that interface with end devices (sensors and control points) using different protocols.

These drivers (e.g. Modbus) are built in a way that made it simple to extend to a new sensor (e.g. a new energy meter) on an already supported physical layer protocol. Collected data gets pushed into a local buffer from where a publishing thread (an independent one from the drivers) would read the data and send it to the cloud stack. This sensing architecture formed the core and efforts in the initial years were made to ruggedize it and augment it to serve several business level use cases such as:

1. Adding new devices on existing supported protocols (e.g. supporting a new energy meter on Modbus)
2. Supporting new devices that communicate on new protocols (by adding corresponding drivers). We have supported BACnet, SNMP, MQTT(s), HTTP(s), TCP, etc. over the years.
3. Adding software components that improved the reliability. Some examples of such services include service with multiple fallbacks for syncing time or addressing the DNS issues at the edge, a service to monitor and manage all other services running on the edge.
4. Building upon the edge intelligence. A simple example is to take raw sensor data, perform some basic computations and create computed time series which is then pushed to the cloud.

On the cloud side, sMap came with a time series DB to store raw data and a relational DB (to store the metadata that does not change with time). Initial 1–2 years were spent in enhancing the devices and query layer that supported different business logics and developing suitable applications that could query and process the data from the database. We added the notion of virtual devices (internally called a “virtual meter” if we were splitting a single device into smaller sub-devices such as monitoring 3 individual phase loads using a 3-phase meter and “pseudo meter” if we were combining data from multiple devices to create a single device). In terms of applications, we focused on the following:

1. Alert Service — This allowed us to process real-time data and send suitable notifications.
2. Reporting Service — It provided a framework to code different analytical logics to then create an output that is either rendered on a dashboard or sent as an email.
3. Dashboard — A dashboard-based user interface layer that kept evolving as we evolved business requirements.

Beyond this, we were also clear from the beginning that we will be deploying hardware at a scale that has to be serviced efficiently. That led to a focus on developing a technology stack for automated testing before the hardware gets shipped and when the hardware gets deployed. Each hardware that was shipped out was tested through automated scripts. With each hardware, we collected a lot of health information as well. As an example with a WiFi-based temperature sensor, we collect RSSI value, reboot counters, reconnect counters that help us in understanding how the hardware is performing in the field (as it gets exposed to different contexts) and hence address corner cases (which are then pushed to already deployed hardware as well using over the air updates). These health parameters, collected from the field, also helped us in intelligently deciding the thresholds for the tests that were being performed before the hardware gets shipped and once it is deployed in the field. More on this when I write in detail on the Analytics vertical.

This relentless focus on developing suitable technologies with a mindset to ensure that the deployed hardware is performing well in the field has helped us scale to a large number of deployments with a complex interconnection of end nodes and gateways at each deployment.

We believe every OEM should think similarly. The data from deployed hardware in the field gives a lot of inputs to how the whole hardware system is to be improved (beyond giving useful application-level information).