Your specialization is what you do: data engineering, cutting hair or cooking steaks. Your niche is who you do it for: aluminum process engineers, investment bankers on Wall Street or cardiologists in France. A niche can also be thought of as a sector (aluminum industry, banking, and medicine), but I prefer to view a niche as a specific group of people you serve.
I was lucky to discover data engineering relatively early in my data career. Still, after fully committing myself to this specialization, I couldn’t stop myself from switching jobs and projects frequently. Often after less than a year. Some roles I left for a good reason: the technology, budget or support was simply not there. Other data roles were perfect on paper: bleeding-edge technology, smart coworkers, flexible working hours, … Yet I consistently grew bitter after six months. Did I have unrealistic expectations? Maybe I didn’t like data engineering after all?
Recently it dawned on me that my specialization was not the problem. I just didn’t like the niche I worked in. As a data engineer, I’m the first in line to deal with the data and its potential problems. Other tech specializations like software developers and data scientists operate more downstream in the machine learning workflow. Once the requirements are defined, it allows these other specializations to abstract away most of the details about the business and focus on their craft. As a data engineer, I’m in direct contact with the stakeholders and must “speak their language”. To understand their data, I must acquire domain knowledge about their specific business.
I realized that in my past I have always applied for jobs based on the required technology. If I liked Airflow, I would look for a company that orchestrates their data pipelines with Airflow. After all, that’s how people build expertise, right? This led me to work in industries that I was not passionate about, to put it mildly. Building an understanding of the data often felt like a chore. I felt it distracted me from what I mistakenly thought was the interesting “technical aspect” of my specialization: ingesting data, data profiling and automation. The truth is: you can’t properly ingest, profile, or automate that which you do not understand.
My current role as a data engineer at Novelis is very similar to the previous roles I abandoned: I’m still working mostly remotely, on a Windows computer or in the cloud, with familiar tools like Python and Spark. Yet, I feel more fulfilled and “in the zone”. What changed?
I picked this role because of the niche and not because of the specialization. I liked the fact that Novelis was a leader in the aluminum industry. It’s an industry I studied at university and where I worked as a process engineer before I pivoted my career into data. I enjoy visiting the huge rolling mills wearing my steel-tipped boots and helmet and having a coffee with the shift workers. They sometimes complain about problems in the process, and I can relate.
Don’t get me wrong – technical expertise is king. Most of my day is still spent behind a computer, often far away from the plant. But having a genuine curiosity and a thirst for knowledge serves me well. I find that when a data engineer is genuinely interested in the business, barriers between teams become smaller and fewer details are lost in translation.
I don’t have all the answers and am still figuring out my own path. But if you’re consistently frustrated by what’s on your computer screen, maybe it’s time to take a critical look at your surroundings instead? Maybe your specialization is not the problem – but your niche.