Slow and steady

We made a lot of progress in our product over the last two weeks. It was a bunch of work, and it took nights and part of the weekend to get where we are right now.

I love the progress, but I hate what we had to do to get there.

The speed we showed is not sustainable. We can clap all we want but after a couple more weeks working like this everyone will be burned out and unwilling to move. This is not a great way to build a product. We are supposed to be running a marathon, not a sprint.

(Funny that Scrum uses the term “sprint” to refer to each iteration. I get why, but I don’t like the implication of speed wrapped in that term.)

We are scaling back the effort a little bit. Not sandbagging, but trying to be a little bit more realistic about what we can do with 80 hours of work.

I’d instead do slow and steady. We are playing the long game here, and my job is to make sure we get there as keen and excited as we started.

The stale backlog

Backlogs become stale quick.

As soon as some of the stuff reported in the backlog stops being relevant, the whole thing becomes useless and distracting.

I just removed 27 irrelevant entries from the backlog of a project I'm working on. There were 27 things reported in there.

I bet you a lunch that if you go to your backlog right now, you'll also find tons of stuff that doesn't make sense anymore. The amount of cycles your team spends every week sifting through stale requests is wasteful and completely unnecessary.

This is why I like to get rid of backlogs. Completely.

Instead, I keep a high-level plan for the next 4 - 6 weeks. If a feature or idea is not ready to be scheduled for that timeline, I ignore it. Important things will find a way to surface again and again, and as soon as there's enough clarity around it and we are ready to tackle it, I'll add it to the schedule.

This helps my team stay focused on what's important. It keeps them away from having to fish what's relevant from an ocean of dead requests.

The fear is always the same: you think you'll forget what's essential, therefore you need to add it to a list somewhere. I've found that this is the least of your problems when developing software. Forgetting is a good thing: it means that you didn't spend time doing something that wasn't important enough.

There will always be tomorrow

I understand some people have a different way to get to the finish line. Mine has evolved. I'm still a work in progress, but it's evident where I stand nowadays: I care first and foremost about putting something —anything— out there, perfection be damned.

I always struggled with convincing stakeholders about progress. More often than not I had to dance around the whole "...but the framework is already in place" idea. I liked to work from inside out; let's make sure we nail the "how," then we worry about the "what."

Not only this makes for a cumbersome conversation with people wanting to see the thing, but it also leads to waste as soon as it evolves.

More and more I try to focus on the product that people see. This is always my trade-off, the way I set my priorities. I focus on the stuff that fuels discussions, and drive ideas. I want products to evolve quickly, and I want as little waste as possible during that process.

There will always be tomorrow, and two hundred oportunities to make the "right thing." Just make sure you win yourself a ticket to fight that battle.

Don't get out of the way

A common saying that goes around is that a good leader "gets out of the way and let their teams do what they do best." I like it. I've also said it before.

There are good intentions packed inside those words. The whole idea is a direct shot at micromanagement and those who continuously become a roadblock putting their unnecessary self in the middle of everyone else.

I get all of that, but I also feel the idea is incomplete and somewhat misleading.

Good leaders don't get out of the way; they get right in front of the action. Good leaders don't disappear; they lead. No matter how good is your team, if they are craving for something, it's probably for your leadership.

There's a difference between leading and telling people how to do things, giving too little room for creativity, or overshadowing everyone all the time. The best thing you can do is not to "get out of the way," but to figure out how exactly you can help progress.

Don't make yourself invisible. Au contraire, you are supposed to be the always-shinning light making the path forward clear for everyone.

Commenting my code

A long time ago I wrote about how I felt about comments in my source code. It was back in 2010, but I'm not going to link back to those posts because I wrote them in Spanish. If you are sufficiently motivated, feel free to search the archives.

The story is that, back in the day, I wasn't very fond of comments. I had this personal quest where I had to refactor my code until every single comment was unnecessary. I still think is a noble mission, but the years have made me realize how hard it is to accomplish.

Add a few years of C development to the mix—where things are as hard to read as ancient hieroglyphs—, and my code has evolved to a completely different stage: nowadays half of my writing is not for computers to compile, but for humans to read and understand.

Time and experience do wonders to people.

There's something that hasn't changed, however: I never write comments about the what, but only about the why.

The what my code is doing should be obvious for any other developers — at least, I try hard. Sure, there are cryptic lines here and there, but most of the time, the code should be enough to the reader. The why however, is a different story: more often than not I try to go out of my way to make sure my intentions are clear through comments in the code, and every time I go back and read what I wrote, the ideas I had become apparent and I feel I'm right there the same day I came up with them.

I'm pretty much addicted to this, so I really think this is the last stage of the evolution regarding comments. I'll get better at writing them, for sure, but I don't think I can go back to noble missions that are hard to accomplish.

Finished

I always wanted to go to a US college, so back in 2015, I started my Masters in Computer Science at Georgia Tech, and just this month of May 2019, after a lot of work and effort from my entire family, I graduated with a major in Machine Learning.

Incredible experience. A lot of sacrifice for sure, but in my opinion, absolutely worth it — not because of the doors that this could open in the future, but because of all of the knowledge that you get.

I'm happy that it's over, but I would do it all over again if I had to.

The question now is, what's next?

Never forget where you came from

Here is kind of a cheezy advice for a new manager, but one that I found quite useful as I got more leadership responsibilities a few years ago:

Never forget where you came from.

Some people get promoted and all of a sudden they think they are a big shot. They stop looking at their friends the same way, or hanging with those folks that "stayed behind." They change the way they talk, behave, and the circles they spend time in.

They fail to realize they are still the same person they were before the promotion. Their inability to act like the person who they are speaks about their stupidity and lack of humility. They forgot why they were promoted in the first place: for being who they were.

If you are promoted, go back and do more of all the shit that made you the person you are today. We need more of that, not less. Remember you aren't unique, and most important, without those people that supported you in the first place you aren't anyone.

Don't turn your back to the rest of us.

Mixed feelings

After four weeks of working full time with AWS, I have mixed feelings about the current state of cloud development. This is a great time to develop enterprise applications in the cloud: AWS (and Azure, and GCP) makes so many things possible — things that were complex before. You can quickly stitch together services and create a sophisticated, integrated system with much less effort than before.

This is great.

But I also feel that there's so much more than can and needs to be done. Over the past month, I estimate that we have spent 90% of our time worrying about infrastructure and only 10% working on the actual code of the application. Sure, at the end this might be for the greater good, but a lot of things have become too cumbersome and time-consuming. Things that are supposed to be much simpler.

Developers can't just be developers anymore; they have to learn a ton of stuff that wasn't part of their jobs before. Or maybe, this is just what being a modern "developer" means.

I'm looking forward to a time where coding your application becomes the main focus again, and the infrastructure piece "just works." We've made a lot of progress, but we haven't arrived yet.

Importing related files from an Amazon S3 bucket using an AWS Lambda function

There's an Amazon S3 bucket that we need to monitor to process files copied into it. Doing this is pretty straightforward by invoking a lambda function whenever the s3:ObjectCreated:* event is triggered by S3.

The Internet is plastered with examples on how to set up this process. You should be able to get everything up and running after a few minutes following one of those tutorials.

But here is a different twist to this problem: we are going to be receiving files in pairs that need to be processed together. Specifically, we will be receiving an image file (let's assume a .PNG file for simplicity) along with related information (metadata) stored in a .JSON file with the same name. For example, we will be getting a file named image1.png and image1.json copied into the bucket, and we need to make sure they are processed together.

Let's define our problem a little bit more formally:

  • Files will come in pairs, a .PNG file and a .JSON file with the same name.

  • Files might be copied at different times into the S3 bucket. We might get the images first, then the corresponding metadata files at a later date, or vice versa.

  • We need to process the files as a unit. We can't handle the image nor the metadata until we have access to both files.

AWS triggers an event for each object created in the S3 bucket, so for each pair of files, we will be getting two separate lambda invocations. We can't make assumptions about the order of the invocations, or how long until both files are ready, so we need to build some synchronization plumbing to take care of this.

The idea behind the solution

I'm sure there are multiple ways to tackle this problem, but we wanted to make things as simple as possible, so we decided to have our lambda collect the files as they show up, and only trigger the processing step whenever we have the pair together.

We can't keep the files in the original S3 bucket because we aren't sure how long it will take for the pair to be ready, and the original files might be removed before we have a chance to collect both of them. This means that we need to copy the files to a separate S3 bucket as soon as we get access to them, and hold them there until we get access to the second file of the pair and have a chance to process them.

To keep track of which file we have and which one we are missing, we can use a DynamoDB table to keep track of where we are. Whenever the lambda function is invoked, we can check the DynamoDB table to determine whether we have both files of the pair, and only move to the processing step when we do.

The radiography of the lambda function

Here is a high-level description of what our lambda function looks like. Remember this function is invoked with every s3:ObjectCreated:* event triggered by the source S3 bucket:

  1. Copy the file from source S3 bucket into the target S3 bucket — this target bucket is our temporal space until we are ready to process the pair of files.

  2. Get from DynamoDB table the record corresponding to the file — we can do this using the name of the file as its identifier.

  3. If a record doesn't exist, create a new one with a status of loading — if a record doesn't exist, it means that this is the first file corresponding to the pair, so we just need to create a new record and do nothing else.

  4. If a record does exist, we can update its status to ready and invoke the processing step.

The code that makes things happen

The gist above shows the Python implementation for this lambda function. Notice that the code makes the following assumptions:

  • Files will be copied in a source S3 bucket that's connected to this lambda function. You can do this by following any of the available samples published as part of AWS' documentation.

  • The lambda function will copy the files to a bucket named temp-bucket. Make sure you change this reference in the code to the name of your own bucket.

  • There's a DynamoDB table created with the name dynamodb-table.

  • There's a (mysterious) invoke_processing_step function that I'm leaving outside of the code. This function receives the name of the file and takes care of processing the pair.

It could be a little bit more complicated

There might be more than two files that we need to process together. In that case, the DynamoDB table will have to store a little bit more information: which files have been read and which ones are missing. Extending the code to support this scenario shouldn't be much more complicated, so I'm leaving that to the reader.

You might also want to remove the records from the DynamoDB table as soon as you finish processing them. I'm assuming this is outside of the scope of this lambda function —or at least, it's not part of the main thread that I tried to follow with this post—, but keep that in mind.

It came out pretty good

In the end, the code came out pretty clean, and the process seems to be holding up pretty well. I'm curious about the results when we stress-test it by bombarding all sort of files into the bucket, but I'm confident things will go as expected.

I'd love to hear about other ways to solve this same problem or any considerations that we might have missed when designing this solution. Don't hesitate to reach out if you have any comments.

Looking past the surface

This one took me a long time to learn.

People applying for an open position at your company are much more than what their resumes say. They are more than their education and experience. They are more than their ability to answer questions under pressure or make a first good impression.

They are people. Unpredictable human beings full of surprises.

Sometimes one has to look past the surface and evaluate candidates based on the impact they can make on your team. Instead of only focusing on them as individuals, think about the whole picture. What could improve if this person starts working with us? How is this person going to change things around here?

A lot of good things can happen when people are giving the opportunity to work under the right conditions around equally motivated individuals. These things are hard to quantify and they usually don't come up in 60-minute interviews, let alone in 500-word resumes.