Writing

Feed Software, technology, sysadmin war stories, and more.

Thursday, April 30, 2020

If you're hiring, be forthcoming about the dev experience

Companies shouldn't be afraid or embarrassed of their dev environments. It is what they use to do stuff, and what every single software person will end up using, so they should be forthcoming about it! This way, if it sucks, people can see it as a warning and stay far away.

Imagine some of the things you could learn.

Company A has you write code on your Mac laptop. Then you start up a large virtual Linux box on the laptop and it NFS-mounts your homedir from the Mac over a virtual interface. It has to download tens of GBs of packages to do this the first time, and it periodically (and unpredictably) downloads large updates. This spins up the laptop's fan. It then starts up "mini me" versions of other services and then starts yours. These versions are not maintained and not initialized, and must have manual steps run the first time or they will silently fail closed and not supply data (like a list of employees). You also can't talk to the real (prod) versions of those services. The first time your code talks to prod is when it itself goes to prod.

Most of the commands you run to do your job happen at a shell prompt on your Mac. Your experience is defined by the Mac UI and window controls.

Company B has you write code on your Mac laptop. You then drop into a continuous build tool and type your username in four times. You click a button and about five minutes later, you are assigned a very large virtual machine running somewhere else - thousands of miles away. This machine does not share any files with your Mac. You can access the Internet from it, but you can't obtain the sources from your company's repos ("git clone ...") since they are hosted with a vendor and rely on ssh auth. Your ssh certs are locked up in your laptop's ssh-agent and would be non-trivial to punch all the way through to the VM. Everyone must run a "hack tool" which will constantly rsync your current project onto the VM, or they must otherwise sync it manually. This VM runs "mini me" versions of other services in addition to your own. They are not initialized, and must have manual steps run the first time or they will silently fail closed and not supply data (like a list of employees). You also can't talk to the real (prod) versions of those services. The first time your code talks to prod is when it itself goes to prod.

Most of the commands you run to do your job happen at a shell prompt on your Mac. Your experience is defined by the Mac UI and window controls.

Company C issues you a Mac laptop and a Linux workstation. The Linux workstation is installed at your desk along with two large LCD screens on arms, a full-size keyboard and mouse. It runs a version of a popular Linux distribution which has been extended with support for the peculiarities of the corporate network and infrastructure. Your workstation primarily runs a web browser and your development environment - X terminals, editors, that sort of thing. You can talk to prod through a proxy, or you can just ship a binary into prod and let it run there -- as you. It can use limited versions of prod services accordingly, like certain read-only features. You can also build a package from that binary and use it to update your real service. This can happen manually, and you can also stand up a continuous build system to produce a new package every time a change affects your code. You still need to manually pick versions to update your production service. Your laptop is used to poke at things from places other than your desk, like in a meeting, or from home. You usually connect to your workstation from it.

Most of the commands you run to do your job happen in a shell within an X terminal on a regular Linux box which is right there at your desk. Your experience is defined by whatever window manager and/or desktop environment you want to run.

Company D issues you a Mac laptop and a Linux development server in a data center within 1000 miles of your location. Your Mac is used for web browsing and e-mail, and you ssh to your development server. The server itself is almost the same hardware as production servers, and is physically near many of them. You run your editor and builds on that host directly. It can talk to production -- as you. It can see limited versions of prod services that you could also talk to. You can build a package on it and use that to ship your service to production. You can also set up a continuous build system to produce a new package periodically. You still need to manually pick versions to update your production service.

Most of the commands you run to do your job happen in a shell inside a ssh session to your development box inside a Mac terminal app on your laptop. Your experience is defined by the Mac UI and window controls.

Company E is like company D: you get a Mac laptop and a Linux development server somewhere within 1000 miles. However, you can get away with running a Linux virtual machine on the Mac. You are allowed to teach it how to participate in the Kerberos scheme, and thus can connect directly from that VM to your development server. Everything else is the same.

Most of the commands you run to do your job happen in a shell inside a ssh session to your development box inside an X terminal on a virtualized Linux box which is on a Mac laptop which is right there on your desk. Your experience is largely defined by whatever window manager and/or desktop environment you want to run. You can still use the Mac native side of things, like by opening the lid of the laptop or by plugging in a second display.

Company F hands you a pile of parts resembling a computer and lets you scrounge around to find someone to burn a CD of your favorite Linux distribution. You are able to use your scissors to back out the latch to the data closet because you know there is a box of Ethernet patch cords on the floor. You put the box together, install whatever you want, plug into the network, do DHCP, start X and a browser, and are able to hit the internal company site and start learning how it works.

Your entire job consists of doing things in the browser and sshing to other machines to do things, so every command happens in a shell in an X terminal on that Linux box right there at your desk. Your experience is completely defined by whatever you decided to load on the machine. You run whatever WM/DE you feel like running that day.

Company G hands you a Linux laptop and puts a BIOS lock on it. It powers up and shows a terrible text-mode prompt for a password. It is a user-mode password, such that you can't get into the BIOS to change things or remove the lock. You are told that if it powers off, you will need to come back to them to have them type it in again. You are not put into group wheel or the sudoers file, so you can't change what is installed on the machine. The fact that your job (uniquely) grants you root on every single production machine to do deep troubleshooting does not influence their decision, and you have to go to lengths to do simple things like installing the window manager and editor you like on the machine. You give up and ask for a Mac laptop, "steal" the user certificate from the Linux laptop, and run your own Linux VM on the Mac laptop using that certificate. You leave the Linux laptop in your desk, totally disconnected. Nobody notices that it hasn't checked in for a year.

Your entire job doesn't matter. Nobody does any engineering there anyway. Attempting to build new things or change anything which already exists is met by extreme pushback from people who control their established domains with iron fists.

I have worked at all of these companies. If I had known about some of these situations beforehand, I might have stopped short during the interview process. I know what works for me and what doesn't.

Companies should be proud of whatever they are running, and not try to hide it. If it sucks for some people, those people will vector around it. Then maybe they'll learn something and try something different if they want those folks around. Meanwhile, the people who do really enjoy that sort of thing will flock to them. Everyone wins.