Supporting a world-class documentation technique with Atlan
The Energetic Metadata Pioneers collection options Atlan purchasers who’ve accomplished a complete evaluation of the Energetic Metadata Administration market. Passing on what you have discovered to the following knowledge chief is the true spirit of the Atlan group! That is why they’re right here to share their hard-earned insights into an evolving market, what makes up your trendy knowledge stack, revolutionary use circumstances for metadata, and extra.
On this installment of the collection, we meet Tina Wang, Director of Analytics Engineering at Tala, a digital monetary providers platform with eight million clients, included in Forbes’ FinTech 50 record for eight consecutive years. She shares her two-year journey with Atlan and the way their robust documentation tradition helps their migration to a brand new next-generation knowledge platform.
This interview has been edited for brevity and readability.
May you inform us somewhat about your self, your background and what attracted you to Knowledge & Analytics?
From the start I used to be very enthusiastic about enterprise, economics, and knowledge, and that’s the reason I selected to double main in Economics and Statistics at UCLA. I have been within the knowledge house ever since. My skilled expertise has been in startups and in previous experiences I’ve at all times been the primary particular person on the info workforce, which incorporates organising all of the infrastructure, creating reviews, discovering concepts and plenty of communication with folks. At Tala, I work with a workforce to design and construct a brand new knowledge infrastructure. I discover that job tremendous fascinating and nice, and that is why I’ve stayed on this discipline.
Would you thoughts describing Tala and the way her knowledge workforce helps the group?
Tala is a monetary know-how firm. At Tala we all know that the present monetary infrastructure doesn’t work for almost all of the world’s inhabitants. We’re making use of superior know-how and human creativity to unravel what legacy establishments cannot or will not, in an effort to unleash the financial energy of the International Majority.
The Analytics Engineering workforce serves as a layer between the back-end engineering groups and numerous enterprise analysts. We construct infrastructure, clear knowledge, configure duties, and ensure knowledge is straightforward to search out and able to use. We’re right here to ensure knowledge is clear, dependable, and reusable, so analysts on groups like Advertising and Operations can concentrate on evaluation and producing insights.
What’s your knowledge stack like?
We primarily use dbt to develop our infrastructure, Snowflake for curation, and Looker for visualization. It has been nice that Atlan connects to all three and helps our means of documenting YAML recordsdata from dbt and routinely syncing them with Snowflake and Looker. We actually like that automation, the place the analytics engineering workforce does not want to enter Atlan to replace the knowledge, it simply flows from dbt and our enterprise customers can use Atlan immediately as their knowledge dictionary.
May you describe your journey with Atlan up to now? Who will get worth from utilizing it?
Now we have been with Atlan for greater than two years and I believe we have been one in every of its first customers. It has been very, very helpful.
We began constructing a presentation layer (PL) with dbt a 12 months in the past, and beforehand used Atlan to doc all of our legacy infrastructure manually. Beforehand, documentation was inconsistent between groups and it was typically obscure what a desk or column meant.
Now, as we construct this PL, our aim is to doc each column and desk that’s uncovered to the tip person, and Atlan has been very useful for us. It is rather straightforward to doc and quite simple for enterprise customers. They’ll go to Atlan and search for a desk or a column, they will even search for the outline, saying one thing like, “Give me all of the columns which have folks data.”
For the analytical engineering workforce, we’re sometimes the curators of that documentation. Once we create tables, we sync with the service homeowners who created the database to know the schema, and once we create columns, we arrange them in an easy-to-read manner and put them in a dbt YAML file, which flows into Atlan . We additionally go into Atlan and add readme recordsdata, if needed.
Enterprise customers don’t use dbt and Atlan is the one technique to entry Snowflake documentation. They’re going to go into Atlan and search for a selected desk or column, they’re going to be capable to learn the documentation, and so they’ll discover out who the proprietor is. They’ll additionally go to the lineage web page to see how one desk is expounded to a different desk and what codes generate the desk. The perfect factor about Lineage is that it’s fully automated. It has been very helpful in knowledge exploration when somebody is unfamiliar with a brand new knowledge supply.
What’s subsequent for you and your workforce? Is there one thing you might be excited to construct?
Now we have been researching the semantic layer of dbt for the previous 12 months. It’ll assist to additional centralize enterprise metric definitions and keep away from duplicate definitions throughout a number of analytics groups within the firm. As soon as we have just about completed our presentation layer, we’ll construct the dbt semantic layer on high of the presentation layer to make reporting and visualizations smoother.
Do you may have any recommendation to share along with your colleagues from this expertise?
Doc. Positively doc.
At one in every of my earlier jobs, there was no documentation of their database, however their database was very small. As a primary worker, I used to be a giant proponent of documentation, so I went in to doc the entire thing, however that might reside in a Google Spreadsheet, which is not actually sustainable for bigger organizations with tens of millions of tables.
Upon arriving in Tala, I found that there was a lot data that it was a problem to navigate. That is why we began the documentation course of. earlier than We construct the brand new infrastructure. We documented our outdated infrastructure for a 12 months, which was not a waste of time as a result of whereas we’re constructing the brand new infrastructure, it’s straightforward for us to discuss with the outdated documentation.
So, I actually emphasize documentation. Whenever you begin is the time and place to actually centralize your data, so that each time somebody leaves, the data stays and it is a lot simpler for brand new folks to affix. Nobody has to play a guessing recreation. It’s centralized and there aren’t any questions.
Typically completely different groups have completely different definitions for related phrases. And even in these circumstances, we’ll use SQL to doc in order that we will say, “That is the formulation that derives this definition of Revenue.”
You wish to go away little or no room for misinterpretation. That is actually what I want to emphasize.
The rest you wish to share?
I nonetheless have the spreadsheet from two years in the past after I seemed for documentation instruments. I did plenty of market analysis, analyzing 20 completely different distributors and all of the instruments I might discover. The vital factor for me was to discover a platform that might connect with all of the instruments I used to be already utilizing, which have been dbt, Snowflake and Looker, and that had a robust assist workforce. I knew that once we first got here on board, we might have questions and arrange plenty of permissions and knowledge connections, and {that a} robust assist workforce could be an enormous assist.
I remembered that once we first labored with the workforce, everybody I interacted with from Atlan was very useful and really beneficiant with their time. Now, we virtually work alone and I’m at all times proud to have discovered and chosen Atlan.
Photograph by Priscilla Du Pérez 🇨🇦 in unpack
