A friend of mine (a software engineer) asked me the other day to explain what I actually do. And I got curious to know what prmpted this questio. He said that for him, the network is a black box. When the network team and the security team start talking in a meeting, he just sits there and watches. He waits for them to reach a consensus, and then he goes back to his code.

Being my favorite topic to talk about, I told him the truth:

A network engineer connects hosts to servers, and makes sure the data travels from one point to another.

That is it. That is the whole job.

This question made me think about how much we swayed from the original purpose of the job, and how much we started focusing on the details forgetting about the main goal.

Then why does it look so complicated?

I tried following the trail to the first network pioneers, before the invention of all the protocols and the techy terms we know today. Every piece of technology in our vocabulary was invented after somebody plugged a cable and hit a wall.

Let me walk you through it. Start with the simplest network in the world.

Two computers, one cable. They can talk. Congratulations, you built a network.

Let’s add a third computer. suddenly you add two more cables and connect it to the original two, still manageable. Your network now has 3 hosts.

Add two hundred computers. Suddenly you can’t find a thousand cables and manage them all. You need a box in the middle so everyone can reach everyone. That box is a switch, and it has to learn who is behind which port. That is where MAC address tables come from. These computers all shout on the same segment, and the noise starts to hurt. So we cut the segment into smaller pieces. That is a VLAN.

One broken cable kills the office. Let’s adds a redundancy switch Great idea, except the frames start going in circles forever and the whole network dies in seconds. So we invented a protocol that finds the loop and blocks one path on purpose. That is Spanning Tree.

How can I reach the networks I don’t own? Nobody can hold a map of the entire world, so routers learn from each other. Inside your own house you use something fast like OSPF. Between houses, on the internet, you use BGP.

Anyone on the internet can reach you. So we put a device on the edge that decides what is allowed in and out. That is a firewall.

Your data has to cross a network you do not trust. So we wrap it in encryption and build a private tunnel over the public path. That is a VPN.

I think you get it now, I could go on all day, and justify each and every technology invented.

So what do I actually do all day?

Concretely, the job splits into four things.

Design. Before anything exists, someone has to decide how many sites, which links, which addresses, how much bandwidth, and what happens when a piece of it fails. Most of the pain in networks comes from decisions made here.

Configuration. Turning that plan into commands on real boxes. Each with its own syntax and its own opinions about how the world should work. A big part of the skill here is not the syntax, it is doing the change at 2 AM without breaking anything that was working at 1 AM.

Troubleshooting. This is the real job. Someone opens a ticket that says “the application is slow.” Nobody knows why. My work is to follow the path of the packet and find the exact place where it stops or slows down. Is the interface up? Is the VLAN there? Does the router have a route? Does the firewall drop it silently? Is the far end even answering?

Most of the time, the honest answer I give is: it is not the network, and here is the proof. That proof is the product. Anyone can say “it’s not us.” Showing where the packet actually dies is the job.

Talking to people. Clients, security teams, system administrators, developers. Half of network engineering is translating between people who all describe the same outage in different words.

Why it feels like a black box

Here is the part I find beautiful. The network feels like a black box to my friend because it succeeded.

The whole design of the internet is layers, and the point of layers is that you don’t need to know what is under you. He writes code that opens a socket and sends bytes. Under that socket, his data is being cut into pieces, wrapped, addressed, routed through equipment owned by companies he has never heard of, in countries he has never visited, and reassembled in the right order on the other side. Some of those pieces get lost on the way, and TCP quietly asks for them again without telling him.

He never notices, and that is exactly the goal. A network you can see is a network that is failing.

If you are a developer, you only really need a small set of questions to talk to us:

  • What is the path between my app and the thing it is calling?
  • Is anything filtering that traffic?
  • What is the latency and is there packet loss?
  • What is the MTU on the path? (This one causes bugs that look insane.)
  • Is DNS actually resolving to the address I think it is? Ask those and you will get very far with any network person.

The one line version

The network is not magic. It is cables, addresses, and rules about who can talk to who.

My job is to keep that path open, and when it breaks, to find the exact point where it broke.

Everything else we argue about in meetings is just us disagreeing about which problem deserves to be solved first, on the same path.