1. Brave new world

Introduction

At the time of writing this, I have already been in SB for around half a year. I joined at July 2022. I still remember feeling really proud having able to join a tech company. I also had a severe Imposter Syndrome, feeling that I only got this job because my friend’s hard-sell-ed me to the Data Analyst team lead. I actually failed the interview twice, lol.

Watching a lot of tutorial videos on tools that I always wanted to learn, but didn’t had a chance, didn’t help in building my confidence either. I’ve always heard friends around me, who are in “proper” Data teams speaking of tools like Airflow, Dbt, BentoML, FastAPI, K8S, Docker, etc. No matter how much tutorial videos I chug, I never had the confidence that I “grasped” these tools.

How did I pass the interview?

To be honest, I’m not that sure either. Maybe I should ask my boss about it soon. All I could remember during the interview is me sharing about my bread and butter task back in umobile, mainly but not limited to:

• Making sure BAU pipeline works. Basically this is just cron + sql scripts.
• Speaking with engineers to build new pipelines. There’s so many tables around, with zero documentation. I wonder how I survived.
• I was the DA-in-charge for Marketing. Hence, I would always need to build/enhance dashboards/reports for them for analysis and new product launches.
• ^ Super important to maintain a good relationship with the Business Analyst. Empowering them to self serve equals giving yourself more time for other stuff. Remember that BA will always know more about what the business is doing. My role is more of an enabler.
• Huge ass migration project from OracleDB to MSSQL Server. It was a nightmare because there were too many moving parts, and bad scoping during early stages of the project.

I guess some of it are quite aligned with what I’m doing currently?

First day in office

1. Amazed at the “Library” concept. Later I learned that most of the books in the main halls are donated by my boss. He’s a hardcore reader. Super impressed, really.
2. Macbook Pro. Wow!
3. Two screens per seat. Wow! Too bad it’s 1080p. 😝
4. Most importantly, the onboarding is chef kiss. The onboarding process is structured in a “visiting Pokemon town” journey, having to complete certain tasks before moving to the next town. Also super impressed with everyone having to pass exams in order to use BI tool, and every BU dataset has its own entrance exam. Damn. Imagine the change managment required to pull this off in a corporate.

After going through the onboarding, it was time to get familiarized with dbt and Airflow. Surprisingly, my past experience with running stuff on cron really helped me out here (of course, also all the available documentations that the team prepared).

Also took me a while to refresh my memory on Git, lol. Haven’t used it since I left Wooppy around 4-5 years ago. Was quite surprised that not many people can use Git CI too. Most just used the PyCharm plugin.

Beginner quest

My first big task was to migrate a currency rate table from “full refresh” mode to “incremental” mode. “Full refresh” means that everytime the table is refreshed, the entire table is dropped and rebuilt; on the contrary, “incremental” means update+insert according to a “unique key”. For this currency rate table, the unique key would be “Date + Currency”. The end goal is: When the table refreshes with new data, if a currency for a date already exists in the table, update it, otherwise insert this new record.

This was a really good task for me to get used to:

• creating new models on dbt
• learning about jinja scripting on dbt (thank God I used it back then in Wooppy for HTML templating)
• scheduling automated tasks on airflow
• writing dbt tests
• git commands