Don't just vibe code your silver layer

Did the title hook you in? Good, listen up. I don’t think data teams are much different from other software teams. Your data engineers support your backend by creating your data warehouse, your data analysts develop the frontend by creating dashboards. The two teams usually establish some common area to translate source data into data ready for reporting. This is why usually data warehouses have three layers: engineers land data in one layer, analysts store data for reporting in another layer and then there’s a layer in between to ensure stability. In medallion architecture, these layers are bronze, gold and silver, accordingly.

Like any other application, your data warehouse has a data architecture whether you designed it or not. You can neglect it, deny it, inherit it from your source systems but it is what you use to answer business questions and build reports. Sometimes, it’s fine to inherit the data architecture of the source system and operate using the same entities. For example, it is extremely likely that your source system has something like a “users” table and your business will have many questions about their users. However, you can choose to redefine the source entities based on your business needs. Your business probably doesn’t care about transactions as much as types of transactions, such as subscription renewals, refund or chargebacks. While the source system architecture reflects how your software was built, the silver layer should reflect how your organisation thinks.

Now, I don’t see how anyone – whether it’s a code assistant or a consultant – can come in and design new entities in your silver layer without understanding the business. Some of the relevant information lives in docs, slide decks or other artifacts. But usually it lives as a collective understanding between individuals in the organisation, like weekly meetings or as vernacular in the day to day work. For example, your CRM tracks leads but maybe your sales team really cares about conversions. You could argue that these entities are latent in the organisation but are nonetheless important in the everyday operations.

So when it comes to business questions, it is very likely that they will express their questions not in terms of source entities, but in terms of their operations. This is where you, the data analyst, play a critical role of listening to your organisation and mapping their question onto business entities.

You could have a code assistant build the silver layer for you but you will be the one providing the spec because you are the one talking to stakeholders and answering their business questions. By building a relationship with your organisation, you can develop a sense of what matters most. It’s normal to lean on source entities first but then develop your own entities as your relationship matures.

If you get it right, new colleagues will onboard faster. Business questions will cleanly map to tables. Having AI answer business questions won’t just be a pipedream. That’s the payoff for being empathetic towards your colleagues. Your silver layer is a reflection of how well you listened. Don’t just vibe code it.