How Beam Scales Personalized, Embedded Analytics using Dashboards-as-Code
Beam is a software-as-a-service (SaaS) startup providing digital infrastructure for cash assistance programs, such as public benefits (think rental or utility assistance, SNAP, WIC, or Unemployment Insurance), guaranteed income programs, workforce development programs, and more. Our software enables government and non-profit customers to handle application intake, case processing, direct payments to constituents, and (you guessed it) reporting for a variety of cash assistance programs. By combining responsible technology development, human-centered design, data, and AI together, we seek to improve access to the social safety net.
GovTech for Dummies Data People
In GovTech, product development focuses on solving problems related to the delivery of government services. Government programs often rely on reporting for myriad key operations related to service delivery: financial reconciliation, data-driven program design, program integrity analyses/insights, program impact reporting, communicating with stakeholders, and sometimes even organizing ongoing operations. As it is our responsibility as product developers to be in tune with the needs and challenges of our customers, let’s start there. In broad terms, our customers want a solution that enables the above activities, with a few additional requirements:
Clean and relevant data and off-the-shelf reports
A single source of truth for program data, which can be shared with members of the same organization freely and easily, but with access controls to prevent leakage
Support for multi-tenant environments, especially where a single, large organization oversees many, smaller organizations who all need access to pieces of the data puzzle
Reports customized to their use case and collection practices, but also the ability to establish, share, or utilize best practices for data collection and analysis across programs
Self-service functionality (i.e. the ability to build, save, share their own reports from raw data)
Observability and security
For our part, as a SaaS provider, we also have a few requirements:
We need our products and features to scale. Scalability and customizability are often at odds, especially in the world of BI.
Being a startup, we also have significant resource constraints, especially analyst time. Thus we also need our solution to generalize reporting content as much as possible, capitalizing on our experience in different verticals and allowing us to propagate best practices as we learn them.
And finally, we want to be able to learn and iterate on our solutions over time without breaking experiences to ensure that we’re providing the best possible solutions.
How it Started
Prior to the work described here, we used a product called Domo. Domo is a fully-featured analytics and data platform providing a range of different tools and services. However, its embedded analytics features are customizable only to a standard (i.e. minimal) degree. One can define a data source with row-based access control (RBAC), enabling customers to safely receive cuts of a common dataset. However, if any customer needs anything custom (as customers often do), they need a completely new data set to be stored on Domo servers and any analytics resources have to be hand-built on top of that. As with tools like Tableau, which are highly customizable, one can copy existing resources and manually swap out dataset references to make that a bit easier. At scale, though, such a process makes our palms sweaty (i.e. introduces significant operational risk).
Many BI products nowadays have reporting APIs hacked in, allowing you to interact with certain aspects of the reporting process. However, for many legacy BI tools, these features were targeted at legacy customers who have dedicated BI engineering teams to build applications around a REST API. Not super friendly to a small, scrappy analytics team!
In addition, because we sometimes collect potentially sensitive constituent data on behalf of our customers, we are leery of piping data in and out of too many other companies’ servers. Rather, serverless options that sit on top of our existing stack would meet our needs better.
How it’s Going
Enter dashboards-as-code. Dashboards-as-code (DaC) lets you define analytics resources in formats like yaml or json. A simple idea with a lot of possibilities!
DaC vendors, such as Hashboard are built around code-based configurability from the start, leading to a more natural-feeling API experience. In addition, some companies in this category (HB included) design their APIs/CLIs with analysts and small data teams in mind. This makes it easier for a team like mine to invest in a Python codebase around our BI toolset that our own analysts will enjoy contributing to and can maintain long-term on their own. Our solution is based in Python, using json configuration files, with simple BigQuery and GCP Secret Manager integrations - comfortable tools for the right analytics team.
In line with our requirements, we kicked off our implementation with a few basic principles:
Decouple development from deployment, enabling satisfying analytics workflows that feel more like CI/CD
Our solution must enable us to administer mixtures of custom and boilerplate content at the same time to the same customer
Adding or updating boilerplate resources or including customized content in a customer’s dashboard should be doable with a single-line (or minimal, at least) code change
It should be minimally more difficult to set up a multi-tenant customer than an individual customer
As a starting point, we built out several modular analytics templates we call “boilerplate modules”, to serve as building blocks for a customizable dashboard based on our customers' needs. This modular design allows customers or our internal staff to pick dashboard components from a library of resources, thereby including only the most relevant data for their program, as well as arrange the dashboard in a way that makes sense for them. Leveraging dashboards-as-code and automation has freed up our team's time for development. This efficiency has allowed us to focus on building out more and more modules to meet new customers' varying needs, further enhancing the customization capabilities of our dashboards. In essence, this approach offers the benefits of customization without the time and resource investment required for truly bespoke dashboards.
Here’s the basic workflow structure when we bring on a new customer:
An analyst receives a request for a report. Let’s say we have some boilerplate modules that cover 80% of the customer’s needs, and the remaining 20% requires another small data model (query) and a few views (charts/tables/etc).
The analyst adds a customer entry to the main config file, then runs a script that sources template yaml files into a new customer directory in GitHub, including files describing any relevant, programmatic variables in our application db (streamed into BQ). The script also calls the Hashboard CLI to initiate preview and deploy steps for the analyst to review and publish the new content.
Once the boilerplate files are validated, the analyst adds custom content into a subfolder under the customer directory and re-runs the publishing function. The function sources the custom content into the existing dashboard and initiates the preview and publish steps.
When the project is ready to go, we simply invite the customer’s report users to the project.
And there you have it! Our analysts can now reliably set up a new customer in minutes, with only a bit more effort to customize. For multi-tenant customers, the script simply loops through a list of new customer directories and generates each tenant’s reports, saving a LOT of time (imagine copying and editing a Tableau dashboard 80 times for one contract!).
Conclusion
We like the dashboards-as-code framework for the high degrees of configurability, observability, generalizability that are intrinsic to the framework and for the creative opportunities it provides. Adopting the dashboards-as-code framework has allowed our small team of analysts to focus on what matters most: building clear, concise and reliable dashboards that surface key metrics and empower our customers to make data-driven decisions about their programs. With the reusability of code and ability to automate steps in the process, the framework also frees up our team’s time for further analysis and development.
As we continue to build out our suite of dashboard modules, we plan to develop a self-service, internal administrative tool to enable our partner success team to design and edit customer’s dashboards by simply selecting and reorganizing the modules a customer needs. We are also exploring how dashboards-as-code can help us automate and/or standardize other data science and analytics tasks, such as monitoring machine learning systems.