Before the Adventure Begins

I have noticed that as people get more experience the way they think about projects changes, and their considerations change. I share a string of thoughts about a project I will begin in the near future in order to shed light on some of the considerations that come up before leaving the Shire.

I want to stop building “small sites” in php for a myriad reasons; enough in fact to write a whole post and a half, which I have since deleted because they’re besides the point, and all specific to me and my circumstances.

The thing is, I want to sub in Kotlin for that purpose. I’m seeking to take the things that make small site development in php so easy for me and make them happen with Kotlin.

Now, why Kotlin is another subject that could span a post and a half. And I have also deleted those. Let’s just say that I’m seeking to write most of my code in Kotlin if I can help it.

The goal is to have Kotlin-based websites that:

  • Can be started trivially. A local setup no more complex than configuring an IDE to deploy targeting a server we have SSH access to.
  • Can have a deployment target trivially spun up (just like you can spin up LAMP images in your cloud provider)
  • Can be deployed trivially.

I have gained some knowledge that will make this quest feasible for me:

  • I have become acquainted with many parts of the Spring Framework. Of particular interest for this scenario are Spring JPA, Spring Boot, Spring MVC, and Thymeleaf.
  • I have spent some time making it easier for me to make deployments when I have ssh access to a target: both through the Gradle SSH Plugin and through Exec tasks that delegate to cmd, ssh, and sftp (these are something of a no-no, since they’re not portable at the time and rely on a Windows installation).
  • I have done due diligence making the Spring Boot executable jar, which runs an embedded Tomcat, work as a systemd service; that’s not too much trouble, really, but it’s important to know it’s possible.
  • I have configured a bundle of nginx instances reverse-proxy requests into my running service. Again, not black magic. It’s good to have it down pat to a tedious, repeatable task.

Oh, yes, and why the nginx reverse-proxy into my service? One massive reason is nginx is many better proven to work correctly than my code. I’d rather not bear the privilege of binding to port 80. And running my web services on port 5000 by default makes it easier to migrate into other manners of hosting.

What will I do to accomplish my goals? Probably write a small piece of Kotlin to aid in provisioning the servers, and rewrite my existing Gradle Exec task so that it works one way in Windows and another in Unix-like environments (Gradle SSH doesn’t play well with Gradle’s Kotlinscript DSL, and I don’t see why I would want to have Gradle code in Groovy when Kotlinscript can make it rain). I may wind up creating another Gradle SSH plugin which doesn’t keel over when inserted into a kts configuration.

The piece of Kotlin to aid provisioning will likely have a cloud-specific API client, and ideally would be able to configure the whole environment with a JVM based ssh client. One alternative I would rather not incur involves creating a Custom Image – it would need to be maintained.

The Gradle task to deploy – and in fact the whole project setup – will likely become a template from which to start all the projects. An archetype, if you will, à la Maven.

I’d be cheating a little because in practice part of the code would be shell script that sets up the system. I can live with that.

Why not make it all a shell script? Because complex decision making in a shell script makes me sad.

After mulling this a bit more, and consulting with colleagues about their experience, the adventure will start.

Code is Part of the Machine

I intend to share with you a realization that recently crystallized around discussions related to software complexity and stack comparisons.

Computers are very complex lately. They have multiple processors with multiple cores each, communicating with each other to let you wait for idle seconds while that spreadsheet software opens. Or while the kitty “gifs” (likely rebranded mp4’s) load.

Processors can also have microcode, and on top of them sits an operating system, on top of which is our stack, on top of which is our code.

Computers, with all their hardware and software, are machines that imitate other machines – often better than those that inspired them. For instance, almost any text editor in use today we can think of as an impossibly powerful typewriter.

The code we write, and the code we build upon, are part of the machine we are putting together. Yet this is one of the aspects that need to be taken into account when building stuff for the long run: the amount of code present in your production machines is part of the machine.

Complex machines are harder to reason about, keep running, and upgrade than simpler ones.

There are other things to consider, of course, but with the simple comparisons out there of how different programming languages can be more or less complex I think we need to separate things and think about each aspect individually.

Even though this

print("Hello, World")

is easier to read than this

#include <stdlib.h>

int main(int argc, char* argv[]){
    printf("Hello, World");
    return 0;
}

All else being equal, the latter is by far the simpler machine.

I’m not certain whether I’ll get to develop this idea as much as I want to in the future, or write about other aspects that need to be considered when you’re building long term systems, so here are some related thoughts:

  • All the code in the production computer, no matter where it comes from, is part of the machine you’re building.
  • All production computers that interact in order to make the system you’re building work are part of the same machine.
  • Yes, this includes the client’s computer, and the client’s phone, with all the apps that are running in the device.
  • Routers and switches are computers.
  • Now that I reflect upon it, I think this is something that Microsoft has known for ages, and part of the reason stuff running on MS systems keep working for relative aeons through upgrades. Some of Raymond Chen’s blog, The Old New Thing has convinced me of that.
  • Ease of writing code doesn’t mean ease of maintaining code doesn’t imply ease of running the code.
  • The most valuable thing to come out of DevOps is the very sane notion that the development team should be involved in making sure the system keeps chugging along.
  • Considering the way early digital computers worked and were programmed, thinking about modern computers as “machines that rewire themselves continuously on the fly” becomes very attractive.
  • The total number of possible machines your production computer can become is part of the complexity of your setup, and is a corollary of the idea that code is part of the machine.
  • Whether your dependencies will become updated or not, the interfaces you depend on die or not, or there are vulnerabilities in your code are a different issue.
  • Sometimes we come close to acknowledging that code is part of the machine. In those instances, we call what we’re building a “platform”.

If we’re going to reason about all this, it’s important to not muddle our thought process and address as much or as little of the environment of building and running applications as we really mean to.

There is more where that came from, but I don’t know that I’ll have the tools to write it properly. So there we go, I hope this tickles your mind and gives you a new way to think about software systems.

Individuals and Team Performance

Recently, Tim Ottinger posed a question on Twitter regarding the change
in teams and the impact of the change in their effectiveness.

This made me reminisce about some things. I’ve rewritten this post a few times, unsure of how much detail to go into, or whether to publish this at all, but I think at least a few remarks bear mentioning.

I have seen people who accelerate every team they join, and people who put the breaks on every team they join (yes, there is an in-between, but the extremes prove a point). I’ve seen teams swapped whole and the results become “permanently” faster, and I’ve also seen them become slower.

I think the direction and magnitude of change comes down to a few different factors: how the individuals interact with the craft of software development, how they interact with other team members, and how the organization interacts with the software projects they own.

People who respect software development as a craft, would feel embarrassed about making avoidable mistakes and like efficiency in general tend to accelerate software development when they’re included in a project.

People who make their teammates feel comfortable, are willing to learn, teach, and grow together, and don’t feel ashamed of having to learn new things tend to accelerate software development when they’re included in a project.

Organizations with sane rules, specially about developing internally transferable skill sets in collaborators, knowledge sharing, and architectural uniformity tend to suffer less with team changes.

The opposites are opposites, and there’s a lot of nuance that doesn’t fit in this margin, and I can’t claim to comprehend everything completely.

Managers tend to have an big impact in their teams, which should be obvious, but feels worth mentioning. A supervisor who doesn’t know their team, doesn’t teach, motivate, cultivate, and prune their team, makes the company (and the projects the team is responsible for) more fragile.

Teams are bound to change. Requirements change, people move on, people move in. Having every change in the team be a possible cause to a “permanent” slow down in development is a mind boggling, yet alarmingly commonly accepted proposition.

Software development should happen in a way that a change in personnel is more like going trough a bumpy section of the road and less like slashing a vehicle’s tires.

I have sometimes felt that some companies try to treat their employees as exchangeable. This is a liability. Everyone is different, and software should be written in such a way that unique people can bring their unique strengths to the table. This implies, as in any kind of team, that taking care of the composition is crucial. And this goes not only for skill level and skill type, but for attitudes and tendencies. Compare with team composition in your favorite team sport.

*Permanent is in quotes because … well, obviously I’m not looking into eternity. I mean over a period of time when the team has settled into their “day to day performance level” plateau.

Mid year stream of thought

It has been a while since I have written anything longer than two tweets. I notice that I have over ten drafts I’ve not looked at in months, which probably means that they’ll never be published.

I’ve distanced myself from the local developer scene in the Dominican Republic. I feel ambivalent about this, but as it’s been about keeping my head down and working, and I’ve been doing just that, I feel content. I look forward to taming my most demanding endeavors in the near future and reacquainting myself with my local peers.

On the other hand, I’ve been interacting more with developers from overseas. It has been fun, being more exposed to developers from a different culture and using different technologies. Usually my work would involve people from either different culture or different technology, not the combo I’m enjoying right now. More data points to learn from.

My dear friend Lois has been posting poems on her site since around August 2017; I think some from 2018 were lost, sadly. I look forward to the 2nd anniversary of getting new poems every Friday.

Now, the hum of the servers quiets down, and I have nothing else to wait for. Back into the fray.

Before stopping I want to record I felt happy with Nintendo’s E3 announcements; that it’s been crazy juggling around a dozen projects (I don’t want to stop and count, really), but it’s been very fulfilling and fun to push the boundaries. Oh, and that it’s very important to have controlled temperature for work. No HVAC means lost efficiency, lost happiness, wasted time, and a general discomfort about existence when you have to work intensely.

A Desperate Effort (follow up)

Sustaining an increased pace of work (related to your usual baseline) over long stretches of time is grueling.

If it is so different from the usual that it’s a desperate effort – well, I’ve not been able to sustain it as well as I’d’ve hoped. I have managed to tweak my work style to better fit my desire for more productivity though:

  • I have lightweight tasks – pleasant to start and carry out – peppered throughout the week which still help me move forward
  • I read stuff written by smart people, which motivates me to work
  • I read stuff written by people with a lot more money than I have, which motivates me to work smart
  • I read stuff written by people with a lot more output than I have, which motivates me to work hard
  • I have a sharper line between work and play

This has allowed me to move in the general direction I want to go. Less waste, more productivity, more happiness.

 

A Desperate Effort

My twitter bio states: “[I] live where you vacation, and work the way you play.”

Now, I live in a beautiful tropical country and have no plans to change that anytime soon. But I have been thinking about the latter clause.

I already play play. Hanging out, boardgames, books, videogames… nibble by nibble, a wide spectrum of entertainment and feel-good activities.

Why should work also be play?

Well, we must make a living somehow; we might as well make it fun. That’s the logic behind the idea – and I happen to have a line of work I find entertaining. It’s also lucrative – so it stands the whims of starting and stopping… of not taking things completely seriously. I goof off and coast through it.

It’s living the dream. And I felt proud of it. Hence the twitter bio.

And yet, upon reflection, something felt off. The path to my present train of thought was circuitous. Two crucial stops involved a high workload I took on, and remembering this article about extraordinary effort by Eliezer Yudkowsky.

Let’s take a walk.

Working provides sustenance. Leaving wealth on the table is an option, of course; the decision to work less than you can is tied to the hierarchy of values in life. Like most other decisions.

So – what does it say about my values that I work as play? Well, it says I feel very safe about the future; one of the uses for excess wealth Today is an insurance policy for Tomorrow. It says I don’t have much ambition, or any big plans – nothing that would require a solid chunk of cash to make happen.

Well, crap. That isn’t me. Or – it isn’t who I want to be. A clearer explanation:

See, I see the future as a potentially bleak place. I want insurance against a wide range of conditions that can go wrong… and seem to be going wrong.

I see the present as a bleak place. I agree with jaibot‘s name for his blog: Almost No One Is Evil. Almost Everything Is Broken. I’d emphasize: Broken Badly. Lots of things are going wrong right now.

If for some reason you’re not convinced of that: check out WHO’s 2017 world hunger report, reporting 815 million of people living in hunger; of those, there are over 200 million children; 45% of deaths for children under 5 are related to malnutrition; people who suffer malnutrition as children carry consequences for life, both physically and mentally which impact the economic potential – thus making the next generation vulnerable to the same situation (source: Food and Nutrition Bulletin Vol. 20, no. 4, from the United Nations University).

So. Yeah. Next point would be what difference could I make. The answer is: it depends on what I do. I’m all for a healthy placement of the locus of control. A back-of-the-envelope calculation tells me I could easily save a few kids from early life malnutrition even if my ideas about food production, water purification and affordable housing don’t pan out.

Except I work the way you play – I relax, I have a comfortable workload most of the time, and work mostly when I feel like it. I am acting as if I didn’t have grand ideas and ambitions, and that is embarrassing. So much so that I hesitate to write this down.

So I decided to flip myself upside down. What I want to do is very scary:

“First say to yourself what you would be; and then do what you have to do.”
– Epictetus

So we circle back to Yudkowsky’s post about extraordinary effort. Extraordinary effort implies doing things as if your life depends on it.

I’m not sure that anything less than an absurd effort will be enough to change from my current path into the path I want to be in. Part of it is the fact that this realization on how my current actions differ from what I think would be most virtuous didn’t cause the emotional train wreck I associate with successful conversion into a cause.

What’s the plan, then? Well, in my working hours I will not stop working. With the exception of immediate threat of death. Thirsty? Wait. Tired? Poor thing.

I intellectually recognize that no effort is enough. I will leave it circumscribed to my working hours so I will be reasonably sure it won’t affect my health.

So when I work, I will work. My hope is to work so hard I get sick of it by the end of the day, every single day. I’ll try and do this for three months, then check the results. It should be measurable in cash.

The start won’t be gradual. The experiment is timeboxed – I will do this during work hours (8 hours a day), and during work days (5 days a week, except for some already planned outings). I will check my results on June 24th.

If you read the Yudkowsky article, you’ll notice this effort is closer to the “desperate” side of the scale, rather than the “extraordinary” one. I think I am setting up the base needed for the extraordinary work.

I can already feel the weight of this commitment. And I’m guessing the emotional train wreck will happen in the middle of the marathon. I will try to apply stoic principles to avoid suffering – even when in painful situations. We’ll see how things go.

Brief Reflection on the New Year

The new year, as Neil deGrasse Tyson pointed out in ’15, is cosmically insignificant. Which means there’s nothing inherently special about December 31st, or January 1st.

I’d reached that conclusion myself perhaps a dozen years ago, give or take two; so many festivities and landmarks in the calendar make little to no sense when thought about from a greater standpoint.

The magic of many of these landmarks vanished for me.

From a few years ago, my wife’s asked and made me think about goals for the year. Which has been disconcerting for me – I didn’t do that, and the year turning is not really “a thing”. I wound up having rolling kind of goals, more than year-long goals. “By March I will achieve X”, said me, circa November – and other similarly non-calendar-fitting goals. This was a good idea.

“There are neither beginnings or endings to the turning of the Wheel of Time. But it was a beginning”.

– Robert Jordan, The Wheel of Time

I got to choose my beginnings and my endings, which is liberating, realistic, and consistent with reality – people choose things to do all the time.

But if I could write a letter to dozen-years-ago-give-or-take-two-me, I’d say this:

Much like gravity isn’t very important for microscopic beings, cosmic significance is not all that important for people. We’re not working at that scale. We’re working at a socio-cultural scale, a strong context of a few decades with an ever-weaker context bounded in a few millennia.

In that scale there are many important landmarks, and in that scale you can still make your own landmarks, and those still make sense. They work not even because of historical significance – they work because they’re shared context. The landmarks are a ritual by a group or individual, and have meaning bestowed upon them by the minds realizing them.

The origin is not the most important thing, which is what I used to think. The future of the landmark – how long will it be celebrated? – is not the most important thing.

What do we give up, and what do we gain? What are we setting ourselves up for? That’s the strength of the landmark. That’s its use. Now go, younger-me, and think about this for a time in that sleepless New Year’s morrow, while everyone around you is in oblivion and realizations of how different you are from people around you are gnawing.

The landmarks, they can make you belong if you want. They can set you apart if you want.

Happy new year.

Retrospective: Live Code Streaming 2

Today we built up on the previous progress (see what it was here)

I had my twitter app credentials stored somewhere they’d not be streamed, and developed a bit of code to read and use them without showing them on screen. It’s important that those data don’t make it into a code repo! Afterwards, I used the tweepy API to create a stream following a “track”, which is just what we need to get all publications of a particular hashtag.

What remains is to really flesh out the twitter client, make it store the relevant data (see what data is really available), and get what I need.

It seems it won’t be too long – and it’s been longer than needed because I’ve been explaining things.

If you’re reading this, I want you to know that the process has been with trial and error; we found a bug in tweepy’s documentation (it’s fixed in master, which is not published yet), for instance, and I had to work out how to do things properly.

I invite you to join me on the next live stream, I will write on my twitter (@iajrz) when the date is defined.

The lives we project

When using any form of communication we choose what to highlight or not about ourselves.

Social media is heavily skewed towards people having lives full of peaks, be it really good (mostly) or really bad times that can be empathized with.

This is not new – this also happens whenever there’s any kind of meeting happening aftera long time without seeing certain groups of people… be it people you studied with, or distant relatives.

Unlike before, we’re positively bombarded with that kind of phenomena now – the price we pay for being ever more connected, ever better able to communicate with little regard for space and time, and with very little cost. This is a superstimulus.

This is likely to have amplified emotional impacts – if you’re happy with yourself, you’re bound to be happy for other’s successes. If you’re insecure, then not only will you have to see that one person whose life you thought was certainly not going to be more amazing than yours at that rare meeting… you’re going to be hit time and again with that kind of slap, and it’s not hard to see how we’d be left reeling.

Often the young can be most insecure, but also, at this point, the young are heavy users of social media… and while projecting better or more interesting lifes than they have (by choosing to highlight the good times), they can trigger an attempt to compensate from other people, creating a vicious cycle.

Perhaps turning off our social feeds and focusing on chats would be better? Or parhaps we should narrow down our social circles. In any case, a good way to break this is to stop skewing the way you present your life – in which case stopping to do it may be the least painful way to do it. Or trying to feel better about ourselves would make it more joyous to see someone else be absolutely killing it out there.

In any case, thinking about this and taking a grip on the situation may help you feel better and have improve your days, raising your quality of life.

Projecting Conciliation

In some cases getting someone to think in a way similar to yours or changing your mind is especially important. Besides the usual value for your correct reasoning, there’s an immediate decision at hand which is directly dependent on the result of a conversation or argument.

Being non confrontational is really useful in these situations, but requires particular sharpness and preparedness. You need to understand the issue as well as possible, and a quality I can’t quite defined but have observed recently… it’s a mixture of believing in the best cognitive intentions of your counterpart, as well as trusting their smarts and being genuinely curious. There may be other factors that further help you get into the right frame of mind.

The behavior I’ve observed in people who work like this – I’m not particularly good at this approach, although I’ve somehow pulled it off on some rare occasions – is an apparently open but directed curiosity which drives everyone into the meat of the business and enlightens everyone… and is equally likely to change anyone’s mind, except that the driver, the one projecting conciliation, has a deep understanding of the issue at hand that makes them especially likely to be closer to a good answer to the problem at hand.

This kind of attitude, which is also very calm in the conversation, is the best I’ve seen to steer important meetings in the right direction – whatever it might be. It costs a lot of effort, but practice makes master… perhaps doing it in non-crucial subjects is a good idea from time to time.