Recently I have been writing about the responsibilities of a data engineer and sharing my advice on how to succeed in this job. In the first post, I listed the technical skills that data engineers have. Last week I wrote about the notion of a 10x engineer and how choosing the right team is one of the most important decisions you can make in your career. In this post, we will talk about non-technical (or "soft") skills.

The data engineering department in the company where I work has grown from 3 engineers to 14 in the last year. During this time, I have done many interviews as a hiring manager. Growing a team is always a challenge, but it is also an opportunity to speak to engineers from other companies and find out what makes an engineer successful in their work. After several conversations, some common ideas emerged. Next to technical expertise, the best engineers possess the skills that amplify their impact in the company.

The Elder Scripts blog covers the opportunities and challenges that software engineers and data engineers encounter. I am sharing insights gained from years of work in the industry as an engineer and as a manager of engineering teams. Subscribe to get notified of weekly posts if you haven't already.

Why soft skills matter

These skills I mentioned are often labeled "soft" to distinguish them from the technical, "hard" skills. In my opinion, it is an unfortunate choice of words. Too often engineers assume that the technical skills are "hard" because they are challenging to acquire. In contrast, the "soft" skills seem easy and almost not worth practicing. This idea is pretty far from the truth.

As I see it, the word "hard" refers to the fact that technical skills are easy to estimate. You either know how to write a SQL query, or you don't. Of course, there are nuances, and writing efficient queries takes knowledge and judgment, but at the end of the day, it is easy to prove that your query works. The computer either does what you expect it to, or it doesn't.

The "soft" skills are a different matter. All humans can communicate, plan, make decisions and convince other people. Being good at these things requires deliberate practice. Improving these skills is far from easy because it is difficult to judge your progress objectively. There is no compiler to throw an error if you don't seize the right opportunities or pick the right people to work with. Despite that, I would argue that practicing these skills will bring you enormous benefits. You will enjoy your work more, and you will be better at it.

Vital non-technical skills for an engineer

Understanding the business

The tools that engineers use are the same in most companies. Sure, some teams prefer Java, and some swear by Python. But it is entirely possible for an engineer to transfer their technical expertise from one company to another. They will perform their duties even if they don't have a clear picture of how their company makes money.

However, "our virtues are most frequently our vices disguised." This ignorance of the company goals leaves the engineer unable to prioritize their work well. They have to rely on their manager for prioritization. In many cases, the motivation to do the work suffers as a result. The impact of the daily tasks becomes unclear.

The most important things to know are what market your company operates in, who its customers are, and which companies are the biggest competitors. Does your company publish quarterly updates on the financial numbers? This information will give you a good view of the current state of affairs and the company's goals in the near future.

Building a personal network in your company

This skill is directly related to the previous one. Most of us work in teams, and we communicate the most with our teammates. But suppose an engineer limits their connections in the company to just their team and their manager. In that case, it hurts their ability to understand the broader goals of their projects.

This is especially true for data engineers because our work is intertwined with that of data analytics, data science, product development teams, and others. In a way, data engineering serves as the glue between these other departments. This is why data engineers need to understand the priorities and challenges that other teams face and the new development they are planning in the future.

In many companies, teams have their information pages in internal knowledge management systems. It is a good idea to understand your company's organizational structure and, most importantly, the structure of the teams with which you collaborate the most. Invite your collaborators to informal one-to-one chats over lunch or coffee. Try it a few times, ask about their long-term goals, and I guarantee you will understand their viewpoint much better than if you limit your conversation to project meetings.

Making decisions in uncertain contexts

Some decisions we make in our jobs are pretty straightforward, and it is easy to verify whether we followed the right road. But a lot of questions don't have an obvious answer. You might need to pick a database for a new long-term project, or you are deciding between two offers from data warehouse vendors that look very similar. When you have to choose one option out of many, and you don't know all consequences of your choice beforehand, it is easy to become paralyzed by indecision.

Decision-making is a big topic which warrants its own article or a book. I found that a weighted decision matrix is a good tool in most cases. However, listing the right criteria and assigning proper weights is an art that takes practice.

Taking notes

Note-taking is a real-world superpower for any engineer. We can only pay attention to a few things at a time, and our memory is faulty. On the other hand, our tools become more and more complicated, and the responsibilities of a data engineer seem to grow every year. I deeply admire those unique people who can keep all this complexity in their brains. Unfortunately, I am not one of them.

Since I first began dumping my daily notes into a plain text file, my focus on ongoing work has improved because I didn't have to remember lots of facts and decisions. If it was necessary to recollect something, it was just a matter of searching the big file.

This approach served me well for a few years. Later I added features to the file to make searches more manageable, such as standard date format, links, and hashtags. After a while, I noticed that I was trying to implement a rich note-taking system, so I found one developed by other people that I liked.

It is perfectly fine to keep your notes in a text file, but if it becomes too cumbersome for you, I suggest choosing a system that allows you to:

• Do a full-text search in your notes.
• Create a daily note with a single click.
• Link to other notes in a wiki style.

I am using a system called Roam Research. Other good options are Obsidian or Logseq.

I love to hear from my readers. Please let me know on Twitter or at theelderscripts@gmail.com what you think about the concept of non-technical skills for engineers. Do you have a specific way to practice decision-making? Or do you have a note-taking system you'd like to share?