Developing with LLM

Now that we have LLMs, there are several things that I think is true.

Code is Commodity

This is a bit more complicated than just “it does not matter how something is written or how something is done”. I think this is only true specifically for business logic but not foundational software.

I think something that something like databases and Linux will continue to be written “carefully” and every line of code needs to be vetted.

This is not true for websites, GUI applications, analysis scripts, etc I think the pressure to deliver features will cause the code written at this level to simply grow exponentially.

Because a person can instruct the LLM to implement the high level concept, the generated code itself is pretty much worth nothing. If pressing that button does that it does, then that is all that matters.

Do Every Problems Need To Be Solved

Thoughts on You Are Not Your Job

See: https://jry.io/writing/you-are-not-your-job/

We have to “Be Our Job” exactly because we live in a capitalistic world or any competitive world, really. If we do not embody our job, someone who is more motivated will embody the job and do a better job than you. So if you want to be competitive, you have to devote time and effort into it. Or you can get lucky and be super smart. If you don’t want to be competitive, that is fine too, but then you will get left behind.

The author also laments that we need warmth and empathy. But this is exactly what is missing when we interact with the coldness of LLM and their output. There is no humanity in code that is generated by LLM. There is no story, there is no experience there. People skim the code and pass by without much care aside from its functionality. There is no taste. And yet, it works, and for some, that is all that matters.

Multiplayer Cheating Is A Social Problem

Recently, there have been some discussion in tackling cheating in video games using kernal anti-cheat. This allows the program to run at a much lower level layer of the machine so that it can own and observe the whole system. This is basically giving the program root access to your computer and so it can do whatever it want. In general, you do not want to have other people to own your system.

However, I would argue that does not fix the problem. It misses the point. Even if they have complete control over the machine, what prevents someone from having a coach besides them? What if they recruit slaves to play the game in some MMO gold farm? What if they have better computer and accessories? What if they have better ping? What if they have been playing the game much longer or is simply better?

Ultimately, multiplayer games is a social game, which requires social solution. And a social system works on reputation. Reputation involves social currency. And internet social currency is so cheap to forge, you can simply create new account to mint them.

That is to say, kenel anti-cheat does not actually solve the problem. It simply made backdoors widespread and acceptable.

Phenomenology of Lying (draft)

This might be a bit “deep” but bear with me.

There is an underlying reality that exists beyond our control. To interact with it, we have our own approximation of reality so that we know how to operate or expect what is going to happen. When we communicate with each other, we are communicating this approximation of reality to each other.

Honesty is merely the act of conveying what one’s actual approximation of reality (which is what they actually use to operate on reality).

Lying is merely the act of conveying an approximation of reality that one does not really use.

Understanding lying this way is important, as it gives us a way to really understand what it means to lie. Notice how this definition of lying cleanly excludes ignorance or misunderstanding. As long as one is honest, one is not lying.

A Common Example

Suppose your boss is asking you the timeline for a project, “How long do you think this is going to take?” What should your answer be based on?

To be honest in this instance is to convey how long you think it is going to take under normal circumstances. To lie is to know that the time taken to finish it is probably X, but purposely say it’s smaller or higher than that.

What if you don’t know? To be honest, this is the answer to a lot of questions. When I deeply ask myself how long something should take, I find it tough to give a concrete answer. Is it a year? No. Is it a month? No. Is it a week? No. Is it 2,3 days? Maybe? Is it 1,2 days? Maybe? Perhaps that is what I should have said; it takes 1 to 3 days. But for certain projects, this timeline is not useful. So what should the answer be here? Well, the honest reality is that you don’t know. That ignorance has to be communicated as well. That is what it means to be honest, it is simply the act of conveying your approximation of the reality as you understand it as close as possible.

Why Does It Matter?

Let’s take a step back. Why do we care that our approximation of reality is as close as possible to the underlying reality? Why should we care that it is out of sync?

We care because the underlying reality is uncaring and will move along regardless of what we think of it. On one extreme, one can believe that one can survive a bullet to the head, but reality will soon prove otherwise. In a less extreme example, we care because we react and act in the world based on that approximation; for what else do we have?

I Want The Incovenience Of Password

I like the fact that my identity is somewhat tied to a 16 characters text. Right now, authentication (verifying that you are who you are) depends entirely on being able to figure out a password. This theoretically mean that it could literally be anyone. This means that the website or system does not need to know who is trying to authenticate. Therefore by design, your privacy is protected.

However, passwordless authentication requires you to prove who you are

There's Only This

The fact that we are conscious is very interesting to me. There is no reason for “this” to exist and yet there it is. I have always been curious about this fact of existence as it has occupied my thinking for all my life.

I have previously talked about this particular fascination of mine and how lonely it is.

So in hope of finding other curious souls, I have decided to put this into writing. Though as you will see, this does not make any sense. For one, this will not make any sense to people that “do not get it”. From my own experience, there’s a certain of amount “getting it” or “searching” that needs to happen before one will even comprehend what is being said here. The second reason that this does not make sense will be obvious when you get it. So, either way, this will not make any sense.

Additionally, this knowledge is also useless. There is nothing to be gain from this understanding, it may even be detrimental to your mental health. For myself, it caused a great deal of endless seeking and futile attempt at truly comprehending it.

What Are You?

To begin, what are you? Perhaps this sounds like a silly question, but really, what are you? At minimum, after stripping everything else, where, what, who, are you?

When I was a child, I used to stare at mirror for minutes just focusing on the eye. It is endlessly fascinating (even now), that this is “me”. Regardless of what happens, where I am, how I am; this will always be me. I will never be anything else. This hand, this face, legs, stomach, skin, hair, everything.

This experience gave rise to the sense that my body is not necessarily “me”. I could have been someone else. Of course, this does not make sense. If I turned/changed into someone else, I would be that someone else.

I also like to think about what makes me, me. What is the minimum “thing” that I can still call “me”.

If the environment is removed, am I still there?

If the body are removed, am I still there?

If the emotions are removed, am I still there?

If memories are removed, am I still there?

If thoughts are removed, am I still there?

It is hard to imagine that “I” still exists in the last one, and yet we have all experienced what its like to be “in the zone’. We have all experienced dreaming, where you are mostly just following along (unless you are the kind of person that is able to control your dreams).

All this led me to think that the ‘Self’ is a much more primordial concept that exists by itself. That is to say that it doesn’t matter what happens, the Self must exist for anything to happen. If “you” does not exist, nothing would happen. On the other hand, if something does not touch you at all, they might as well not exist. So the primacy of existence is that you exist.

Getting It

Human (Dis)connection

I had a memory from when I was growing up that I am not even sure is real or not. The memory is that of staring at myself in the mirror and realizing that the person in the mirror is “me”. It was a very stark moment of realization for me that “that is me, that will always be me. That’s it”. I remember staring at the mirror for quite some, Of course, I know that other people exist, and that they are living and interacting iamong each other. Perhaps for a long time I never considered that I am … one of the person?

For long time, I don’t really know what to do with this information/realization. It did made very interested in the self and consciousness. Unfortunately for me, there’s no one else to talk to about this stuff so I have mostly been thinking about this by myself. Thus, it has mostly been an intellectual curiosity for me and nothing more.

It wasn’t until recently that I realized a very grim consequence from this realization: That I will always be alone.

The realization is that no matter how close I am to a person, I will always be in my own head and theirs in their own heads. No matter what emotions or feelings that I am experience, I am experiencing it by myself. This other person is someone that I will never be able to fundamentally connect. It is another person, with wholly different and disjoint experience that I will never be able to tap into.

Perhaps this explain why I usually feel sorrow whenever I am with people that are very close to me. It is as though even I am emotionally so close to them, I will forever be by myself in my own head.

Is this strange? Does everyone else feel the same way?

A Retrospective on The Humble Programmer by Edsger W. Dijkstra

The article is freely available here or at the bottom where I have also copied the transcript below for prosperity. It is crazy to think how much foresight he had 50 years ago.

The increased power of the hardware, together with the perhaps even more dramatic increase in its reliability, made solutions feasible that the programmer had not dared to dream about a few years before. And now, a few years later, he had to dream about them and, even worse, he had to transform such dreams into reality!

This paragraph describes why programming, or something like it, will always exists. The more powerful our tools become, the more ambitious we will be. So no amount of AI will fix that human condition. There’s also a similar observation in economics where the more efficient something is, the more widespread its use. For example, when electricity was hard to produce, people only dreamed of using it to produce light. But now that its abundant, there’s desire and ideas to turn everything digital.

It would be absolutely unfair to blame them for shortcomings that only became apparent after a decade or so of extensive usage: groups with a successful look-ahead of ten years are quite rare! In retrospect we must rate FORTRAN as a successful coding technique, but with very few effective aids to conception, aids which are now so urgently needed that time has come to consider it out of date. The sooner we can forget that FORTRAN has ever existed, the better, for as a vehicle of thought it is no longer adequate: it wastes our brainpower, is too risky and therefore too expensive to use.

Feels like he’s talking about legacy languages here cough C++ cough.

The vision is that, well before the seventies have run to completion, we shall be able to design and implement the kind of systems that are now straining our programming ability, at the expense of only a few percent in man-years of what they cost us now, and that besides that, these systems will be virtually free of bugs. […] Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with.

Here he’s either underestimating how hard programming is or overestimating the capability of the average programmer. It seems like he thought that the programming industry would hone and update their tools to eventually arrive at something “perfect”. But even now, we still don’t know what that is. No one really knows how UI should be done, how async should work, etc etc.

He also thought that software’s acceleration will match the hardware’s acceleration. But this is simply not true, CPUs have gotten a million times faster but is our tools getting as fast? There’s still some delay when I type on a text editor. Is our text editor doing more things today compared to text editor of ye old? Yes, but surely not that bad? For example, take the recent Windows Terminal fiasco (https://github.com/microsoft/terminal/issues/10362). TLDR: Multiple billions dollar company product is shit, someone complained, MS says hard to fix only for said person to implement a fix over a weekend.

I now suggest that we confine ourselves to the design and implementation of intellectually manageable programs. If someone fears that this restriction is so severe that we cannot live with it, I can reassure him: the class of intellectually manageable programs is still sufficiently rich to contain many very realistic programs for any problem capable of algorithmic solution.

Here he’s proposing that we use smaller and more confined language with less foot guns. Something like Rust (does not have UB) or any language with a GC really. In the same paragraph, he’s also talking about something like “correct by construction” which is an interesting concept like https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/. He’s also predicting the use of static analysis and linters like clangd and ESLint.

[…] a specific phenomenon occurs that even has a well-established name: it is called “the one-liners”. It takes one of two different forms: one programmer places a one-line program on the desk of another and either he proudly tells what it does and adds the question “Can you code this in less symbols?” – as if this were of any conceptual relevance! – or he just asks “Guess what it does!”.

Feeling a bit attacked, but he’s both right and wrong here. There’s some utility in one liner, especially when it is used with familiar and regular tools. For example using filter in JS is much more concise and understandable than using raw loops with an if-else inside (IMO).

Original Text

The following text have been copied directly from https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD340.html.

The Humble Programmer by Edsger W. Dijkstra

As a result of a long sequence of coincidences I entered the programming profession officially on the first spring morning of 1952 and as far as I have been able to trace, I was the first Dutchman to do so in my country. In retrospect the most amazing thing was the slowness with which, at least in my part of the world, the programming profession emerged, a slowness which is now hard to believe. But I am grateful for two vivid recollections from that period that establish that slowness beyond any doubt.

After having programmed for some three years, I had a discussion with A. van Wijngaarden, who was then my boss at the Mathematical Centre in Amsterdam, a discussion for which I shall remain grateful to him as long as I live. The point was that I was supposed to study theoretical physics at the University of Leiden simultaneously, and as I found the two activities harder and harder to combine, I had to make up my mind, either to stop programming and become a real, respectable theoretical physicist, or to carry my study of physics to a formal completion only, with a minimum of effort, and to become….., yes what? A programmer? But was that a respectable profession? For after all, what was programming? Where was the sound body of knowledge that could support it as an intellectually respectable discipline? I remember quite vividly how I envied my hardware colleagues, who, when asked about their professional competence, could at least point out that they knew everything about vacuum tubes, amplifiers and the rest, whereas I felt that, when faced with that question, I would stand empty-handed. Full of misgivings I knocked on van Wijngaarden’s office door, asking him whether I could “speak to him for a moment”; when I left his office a number of hours later, I was another person. For after having listened to my problems patiently, he agreed that up till that moment there was not much of a programming discipline, but then he went on to explain quietly that automatic computers were here to stay, that we were just at the beginning and could not I be one of the persons called to make programming a respectable discipline in the years to come? This was a turning point in my life and I completed my study of physics formally as quickly as I could. One moral of the above story is, of course, that we must be very careful when we give advice to younger people; sometimes they follow it!

Another two years later, in 1957, I married and Dutch marriage rites require you to state your profession and I stated that I was a programmer. But the municipal authorities of the town of Amsterdam did not accept it on the grounds that there was no such profession. And, believe it or not, but under the heading “profession” my marriage act shows the ridiculous entry “theoretical physicist”!

So much for the slowness with which I saw the programming profession emerge in my own country. Since then I have seen more of the world, and it is my general impression that in other countries, apart from a possible shift of dates, the growth pattern has been very much the same.

Let me try to capture the situation in those old days in a little bit more detail, in the hope of getting a better understanding of the situation today. While we pursue our analysis, we shall see how many common misunderstandings about the true nature of the programming task can be traced back to that now distant past.

The first automatic electronic computers were all unique, single-copy machines and they were all to be found in an environment with the exciting flavour of an experimental laboratory. Once the vision of the automatic computer was there, its realisation was a tremendous challenge to the electronic technology then available, and one thing is certain: we cannot deny the courage of the groups that decided to try and build such a fantastic piece of equipment. For fantastic pieces of equipment they were: in retrospect one can only wonder that those first machines worked at all, at least sometimes. The overwhelming problem was to get and keep the machine in working order. The preoccupation with the physical aspects of automatic computing is still reflected in the names of the older scientific societies in the field, such as the Association for Computing Machinery or the British Computer Society, names in which explicit reference is made to the physical equipment.

What about the poor programmer? Well, to tell the honest truth: he was hardly noticed. For one thing, the first machines were so bulky that you could hardly move them and besides that, they required such extensive maintenance that it was quite natural that the place where people tried to use the machine was the same laboratory where the machine had been developed. Secondly, his somewhat invisible work was without any glamour: you could show the machine to visitors and that was several orders of magnitude more spectacular than some sheets of coding. But most important of all, the programmer himself had a very modest view of his own work: his work derived all its significance from the existence of that wonderful machine. Because that was a unique machine, he knew only too well that his programs had only local significance and also, because it was patently obvious that this machine would have a limited lifetime, he knew that very little of his work would have a lasting value. Finally, there is yet another circumstance that had a profound influence on the programmer’s attitude to his work: on the one hand, besides being unreliable, his machine was usually too slow and its memory was usually too small, i.e. he was faced with a pinching shoe, while on the other hand its usually somewhat queer order code would cater for the most unexpected constructions. And in those days many a clever programmer derived an immense intellectual satisfaction from the cunning tricks by means of which he contrived to squeeze the impossible into the constraints of his equipment.

Two opinions about programming date from those days. I mention them now, I shall return to them later. The one opinion was that a really competent programmer should be puzzle-minded and very fond of clever tricks; the other opinion was that programming was nothing more than optimizing the efficiency of the computational process, in one direction or the other.

The latter opinion was the result of the frequent circumstance that, indeed, the available equipment was a painfully pinching shoe, and in those days one often encountered the naive expectation that, once more powerful machines were available, programming would no longer be a problem, for then the struggle to push the machine to its limits would no longer be necessary and that was all what programming was about, wasn’t it? But in the next decades something completely different happened: more powerful machines became available, not just an order of magnitude more powerful, even several orders of magnitude more powerful. But instead of finding ourselves in the state of eternal bliss of all programming problems solved, we found ourselves up to our necks in the software crisis! How come?

There is a minor cause: in one or two respects modern machinery is basically more difficult to handle than the old machinery. Firstly, we have got the I/O interrupts, occurring at unpredictable and irreproducible moments; compared with the old sequential machine that pretended to be a fully deterministic automaton, this has been a dramatic change and many a systems programmer’s grey hair bears witness to the fact that we should not talk lightly about the logical problems created by that feature. Secondly, we have got machines equipped with multi-level stores, presenting us problems of management strategy that, in spite of the extensive literature on the subject, still remain rather elusive. So much for the added complication due to structural changes of the actual machines.

But I called this a minor cause; the major cause is… that the machines have become several orders of magnitude more powerful! To put it quite bluntly: as long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming had become an equally gigantic problem. In this sense the electronic industry has not solved a single problem, it has only created them, it has created the problem of using its products. To put it in another way: as the power of available machines grew by a factor of more than a thousand, society’s ambition to apply these machines grew in proportion, and it was the poor programmer who found his job in this exploded field of tension between ends and means. The increased power of the hardware, together with the perhaps even more dramatic increase in its reliability, made solutions feasible that the programmer had not dared to dream about a few years before. And now, a few years later, he had to dream about them and, even worse, he had to transform such dreams into reality! Is it a wonder that we found ourselves in a software crisis? No, certainly not, and as you may guess, it was even predicted well in advance; but the trouble with minor prophets, of course, is that it is only five years later that you really know that they had been right.

Then, in the mid-sixties, something terrible happened: the computers of the so-called third generation made their appearance. The official literature tells us that their price/performance ratio has been one of the major design objectives. But if you take as “performance” the duty cycle of the machine’s various components, little will prevent you from ending up with a design in which the major part of your performance goal is reached by internal housekeeping activities of doubtful necessity. And if your definition of price is the price to be paid for the hardware, little will prevent you from ending up with a design that is terribly hard to program for: for instance the order code might be such as to enforce, either upon the programmer or upon the system, early binding decisions presenting conflicts that really cannot be resolved. And to a large extent these unpleasant possibilities seem to have become reality.

When these machines were announced and their functional specifications became known, quite a few among us must have become quite miserable; at least I was. It was only reasonable to expect that such machines would flood the computing community, and it was therefore all the more important that their design should be as sound as possible. But the design embodied such serious flaws that I felt that with a single stroke the progress of computing science had been retarded by at least ten years: it was then that I had the blackest week in the whole of my professional life. Perhaps the most saddening thing now is that, even after all those years of frustrating experience, still so many people honestly believe that some law of nature tells us that machines have to be that way. They silence their doubts by observing how many of these machines have been sold, and derive from that observation the false sense of security that, after all, the design cannot have been that bad. But upon closer inspection, that line of defense has the same convincing strength as the argument that cigarette smoking must be healthy because so many people do it.

It is in this connection that I regret that it is not customary for scientific journals in the computing area to publish reviews of newly announced computers in much the same way as we review scientific publications: to review machines would be at least as important. And here I have a confession to make: in the early sixties I wrote such a review with the intention of submitting it to the CACM, but in spite of the fact that the few colleagues to whom the text was sent for their advice, urged me all to do so, I did not dare to do it, fearing that the difficulties either for myself or for the editorial board would prove to be too great. This suppression was an act of cowardice on my side for which I blame myself more and more. The difficulties I foresaw were a consequence of the absence of generally accepted criteria, and although I was convinced of the validity of the criteria I had chosen to apply, I feared that my review would be refused or discarded as “a matter of personal taste”. I still think that such reviews would be extremely useful and I am longing to see them appear, for their accepted appearance would be a sure sign of maturity of the computing community.

The reason that I have paid the above attention to the hardware scene is because I have the feeling that one of the most important aspects of any computing tool is its influence on the thinking habits of those that try to use it, and because I have reasons to believe that that influence is many times stronger than is commonly assumed. Let us now switch our attention to the software scene.

Here the diversity has been so large that I must confine myself to a few stepping stones. I am painfully aware of the arbitrariness of my choice and I beg you not to draw any conclusions with regard to my appreciation of the many efforts that will remain unmentioned.

In the beginning there was the EDSAC in Cambridge, England, and I think it quite impressive that right from the start the notion of a subroutine library played a central role in the design of that machine and of the way in which it should be used. It is now nearly 25 years later and the computing scene has changed dramatically, but the notion of basic software is still with us, and the notion of the closed subroutine is still one of the key concepts in programming. We should recognise the closed subroutines as one of the greatest software inventions; it has survived three generations of computers and it will survive a few more, because it caters for the implementation of one of our basic patterns of abstraction. Regrettably enough, its importance has been underestimated in the design of the third generation computers, in which the great number of explicitly named registers of the arithmetic unit implies a large overhead on the subroutine mechanism. But even that did not kill the concept of the subroutine, and we can only pray that the mutation won’t prove to be hereditary.

The second major development on the software scene that I would like to mention is the birth of FORTRAN. At that time this was a project of great temerity and the people responsible for it deserve our great admiration. It would be absolutely unfair to blame them for shortcomings that only became apparent after a decade or so of extensive usage: groups with a successful look-ahead of ten years are quite rare! In retrospect we must rate FORTRAN as a successful coding technique, but with very few effective aids to conception, aids which are now so urgently needed that time has come to consider it out of date. The sooner we can forget that FORTRAN has ever existed, the better, for as a vehicle of thought it is no longer adequate: it wastes our brainpower, is too risky and therefore too expensive to use. FORTRAN’s tragic fate has been its wide acceptance, mentally chaining thousands and thousands of programmers to our past mistakes. I pray daily that more of my fellow-programmers may find the means of freeing themselves from the curse of compatibility.

The third project I would not like to leave unmentioned is LISP, a fascinating enterprise of a completely different nature. With a few very basic principles at its foundation, it has shown a remarkable stability. Besides that, LISP has been the carrier for a considerable number of in a sense our most sophisticated computer applications. LISP has jokingly been described as “the most intelligent way to misuse a computer”. I think that description a great compliment because it transmits the full flavour of liberation: it has assisted a number of our most gifted fellow humans in thinking previously impossible thoughts.

The fourth project to be mentioned is ALGOL 60. While up to the present day FORTRAN programmers still tend to understand their programming language in terms of the specific implementation they are working with – hence the prevalence of octal and hexadecimal dumps – , while the definition of LISP is still a curious mixture of what the language means and how the mechanism works, the famous Report on the Algorithmic Language ALGOL 60 is the fruit of a genuine effort to carry abstraction a vital step further and to define a programming language in an implementation-independent way. One could argue that in this respect its authors have been so successful that they have created serious doubts as to whether it could be implemented at all! The report gloriously demonstrated the power of the formal method BNF, now fairly known as Backus-Naur-Form, and the power of carefully phrased English, at least when used by someone as brilliant as Peter Naur. I think that it is fair to say that only very few documents as short as this have had an equally profound influence on the computing community. The ease with which in later years the names ALGOL and ALGOL-like have been used, as an unprotected trade mark, to lend some of its glory to a number of sometimes hardly related younger projects, is a somewhat shocking compliment to its standing. The strength of BNF as a defining device is responsible for what I regard as one of the weaknesses of the language: an over-elaborate and not too systematic syntax could now be crammed into the confines of very few pages. With a device as powerful as BNF, the Report on the Algorithmic Language ALGOL 60 should have been much shorter. Besides that I am getting very doubtful about ALGOL 60’s parameter mechanism: it allows the programmer so much combinatorial freedom, that its confident use requires a strong discipline from the programmer. Besides expensive to implement it seems dangerous to use.

Finally, although the subject is not a pleasant one, I must mention PL/1, a programming language for which the defining documentation is of a frightening size and complexity. Using PL/1 must be like flying a plane with 7000 buttons, switches and handles to manipulate in the cockpit. I absolutely fail to see how we can keep our growing programs firmly within our intellectual grip when by its sheer baroqueness the programming language – our basic tool, mind you! – already escapes our intellectual control. And if I have to describe the influence PL/1 can have on its users, the closest metaphor that comes to my mind is that of a drug. I remember from a symposium on higher level programming language a lecture given in defense of PL/1 by a man who described himself as one of its devoted users. But within a one-hour lecture in praise of PL/1. he managed to ask for the addition of about fifty new “features”, little supposing that the main source of his problems could very well be that it contained already far too many “features”. The speaker displayed all the depressing symptoms of addiction, reduced as he was to the state of mental stagnation in which he could only ask for more, more, more… When FORTRAN has been called an infantile disorder, full PL/1, with its growth characteristics of a dangerous tumor, could turn out to be a fatal disease.

So much for the past. But there is no point in making mistakes unless thereafter we are able to learn from them. As a matter of fact, I think that we have learned so much, that within a few years programming can be an activity vastly different from what it has been up till now, so different that we had better prepare ourselves for the shock. Let me sketch for you one of the possible futures. At first sight, this vision of programming in perhaps already the near future may strike you as utterly fantastic. Let me therefore also add the considerations that might lead one to the conclusion that this vision could be a very real possibility.

The vision is that, well before the seventies have run to completion, we shall be able to design and implement the kind of systems that are now straining our programming ability, at the expense of only a few percent in man-years of what they cost us now, and that besides that, these systems will be virtually free of bugs. These two improvements go hand in hand. In the latter respect software seems to be different from many other products, where as a rule a higher quality implies a higher price. Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with. In other words: both goals point to the same change.

Such a drastic change in such a short period of time would be a revolution, and to all persons that base their expectations for the future on smooth extrapolation of the recent past – appealing to some unwritten laws of social and cultural inertia – the chance that this drastic change will take place must seem negligible. But we all know that sometimes revolutions do take place! And what are the chances for this one?

There seem to be three major conditions that must be fulfilled. The world at large must recognize the need for the change; secondly the economic need for it must be sufficiently strong; and, thirdly, the change must be technically feasible. Let me discuss these three conditions in the above order.

With respect to the recognition of the need for greater reliability of software, I expect no disagreement anymore. Only a few years ago this was different: to talk about a software crisis was blasphemy. The turning point was the Conference on Software Engineering in Garmisch, October 1968, a conference that created a sensation as there occurred the first open admission of the software crisis. And by now it is generally recognized that the design of any large sophisticated system is going to be a very difficult job, and whenever one meets people responsible for such undertakings, one finds them very much concerned about the reliability issue, and rightly so. In short, our first condition seems to be satisfied.

Now for the economic need. Nowadays one often encounters the opinion that in the sixties programming has been an overpaid profession, and that in the coming years programmer salaries may be expected to go down. Usually this opinion is expressed in connection with the recession, but it could be a symptom of something different and quite healthy, viz. that perhaps the programmers of the past decade have not done so good a job as they should have done. Society is getting dissatisfied with the performance of programmers and of their products. But there is another factor of much greater weight. In the present situation it is quite usual that for a specific system, the price to be paid for the development of the software is of the same order of magnitude as the price of the hardware needed, and society more or less accepts that. But hardware manufacturers tell us that in the next decade hardware prices can be expected to drop with a factor of ten. If software development were to continue to be the same clumsy and expensive process as it is now, things would get completely out of balance. You cannot expect society to accept this, and therefore we must learn to program an order of magnitude more effectively. To put it in another way: as long as machines were the largest item on the budget, the programming profession could get away with its clumsy techniques, but that umbrella will fold rapidly. In short, also our second condition seems to be satisfied.

And now the third condition: is it technically feasible? I think it might and I shall give you six arguments in support of that opinion.

A study of program structure had revealed that programs – even alternative programs for the same task and with the same mathematical content – can differ tremendously in their intellectual manageability. A number of rules have been discovered, violation of which will either seriously impair or totally destroy the intellectual manageability of the program. These rules are of two kinds. Those of the first kind are easily imposed mechanically, viz. by a suitably chosen programming language. Examples are the exclusion of goto-statements and of procedures with more than one output parameter. For those of the second kind I at least – but that may be due to lack of competence on my side – see no way of imposing them mechanically, as it seems to need some sort of automatic theorem prover for which I have no existence proof. Therefore, for the time being and perhaps forever, the rules of the second kind present themselves as elements of discipline required from the programmer. Some of the rules I have in mind are so clear that they can be taught and that there never needs to be an argument as to whether a given program violates them or not. Examples are the requirements that no loop should be written down without providing a proof for termination nor without stating the relation whose invariance will not be destroyed by the execution of the repeatable statement.

I now suggest that we confine ourselves to the design and implementation of intellectually manageable programs. If someone fears that this restriction is so severe that we cannot live with it, I can reassure him: the class of intellectually manageable programs is still sufficiently rich to contain many very realistic programs for any problem capable of algorithmic solution. We must not forget that it is not our business to make programs, it is our business to design classes of computations that will display a desired behaviour. The suggestion of confining ourselves to intellectually manageable programs is the basis for the first two of my announced six arguments.

Argument one is that, as the programmer only needs to consider intellectually manageable programs, the alternatives he is choosing between are much, much easier to cope with.

Argument two is that, as soon as we have decided to restrict ourselves to the subset of the intellectually manageable programs, we have achieved, once and for all, a drastic reduction of the solution space to be considered. And this argument is distinct from argument one.

Argument three is based on the constructive approach to the problem of program correctness. Today a usual technique is to make a program and then to test it. But: program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence. The only effective way to raise the confidence level of a program significantly is to give a convincing proof of its correctness. But one should not first make the program and then prove its correctness, because then the requirement of providing the proof would only increase the poor programmer’s burden. On the contrary: the programmer should let correctness proof and program grow hand in hand. Argument three is essentially based on the following observation. If one first asks oneself what the structure of a convincing proof would be and, having found this, then constructs a program satisfying this proof’s requirements, then these correctness concerns turn out to be a very effective heuristic guidance. By definition this approach is only applicable when we restrict ourselves to intellectually manageable programs, but it provides us with effective means for finding a satisfactory one among these.

Argument four has to do with the way in which the amount of intellectual effort needed to design a program depends on the program length. It has been suggested that there is some kind of law of nature telling us that the amount of intellectual effort needed grows with the square of program length. But, thank goodness, no one has been able to prove this law. And this is because it need not be true. We all know that the only mental tool by means of which a very finite piece of reasoning can cover a myriad cases is called “abstraction”; as a result the effective exploitation of his powers of abstraction must be regarded as one of the most vital activities of a competent programmer. In this connection it might be worth-while to point out that the purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise. Of course I have tried to find a fundamental cause that would prevent our abstraction mechanisms from being sufficiently effective. But no matter how hard I tried, I did not find such a cause. As a result I tend to the assumption – up till now not disproved by experience – that by suitable application of our powers of abstraction, the intellectual effort needed to conceive or to understand a program need not grow more than proportional to program length. But a by-product of these investigations may be of much greater practical significance, and is, in fact, the basis of my fourth argument. The by-product was the identification of a number of patterns of abstraction that play a vital role in the whole process of composing programs. Enough is now known about these patterns of abstraction that you could devote a lecture to about each of them. What the familiarity and conscious knowledge of these patterns of abstraction imply dawned upon me when I realized that, had they been common knowledge fifteen years ago, the step from BNF to syntax-directed compilers, for instance, could have taken a few minutes instead of a few years. Therefore I present our recent knowledge of vital abstraction patterns as the fourth argument.

Now for the fifth argument. It has to do with the influence of the tool we are trying to use upon our own thinking habits. I observe a cultural tradition, which in all probability has its roots in the Renaissance, to ignore this influence, to regard the human mind as the supreme and autonomous master of its artefacts. But if I start to analyse the thinking habits of myself and of my fellow human beings, I come, whether I like it or not, to a completely different conclusion, viz. that the tools we are trying to use and the language or notation we are using to express or record our thoughts, are the major factors determining what we can think or express at all! The analysis of the influence that programming languages have on the thinking habits of its users, and the recognition that, by now, brainpower is by far our scarcest resource, they together give us a new collection of yardsticks for comparing the relative merits of various programming languages. The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague. In the case of a well-known conversational programming language I have been told from various sides that as soon as a programming community is equipped with a terminal for it, a specific phenomenon occurs that even has a well-established name: it is called “the one-liners”. It takes one of two different forms: one programmer places a one-line program on the desk of another and either he proudly tells what it does and adds the question “Can you code this in less symbols?” – as if this were of any conceptual relevance! – or he just asks “Guess what it does!”. From this observation we must conclude that this language as a tool is an open invitation for clever tricks; and while exactly this may be the explanation for some of its appeal, viz. to those who like to show how clever they are, I am sorry, but I must regard this as one of the most damning things that can be said about a programming language. Another lesson we should have learned from the recent past is that the development of “richer” or “more powerful” programming languages was a mistake in the sense that these baroque monstrosities, these conglomerations of idiosyncrasies, are really unmanageable, both mechanically and mentally. I see a great future for very systematic and very modest programming languages. When I say “modest”, I mean that, for instance, not only ALGOL 60’s “for clause”, but even FORTRAN’s “DO loop” may find themselves thrown out as being too baroque. I have run a little programming experiment with really experienced volunteers, but something quite unintended and quite unexpected turned up. None of my volunteers found the obvious and most elegant solution. Upon closer analysis this turned out to have a common source: their notion of repetition was so tightly connected to the idea of an associated controlled variable to be stepped up, that they were mentally blocked from seeing the obvious. Their solutions were less efficient, needlessly hard to understand, and it took them a very long time to find them. It was a revealing, but also shocking experience for me. Finally, in one respect one hopes that tomorrow’s programming languages will differ greatly from what we are used to now: to a much greater extent than hitherto they should invite us to reflect in the structure of what we write down all abstractions needed to cope conceptually with the complexity of what we are designing. So much for the greater adequacy of our future tools, which was the basis of the fifth argument.

As an aside I would like to insert a warning to those who identify the difficulty of the programming task with the struggle against the inadequacies of our current tools, because they might conclude that, once our tools will be much more adequate, programming will no longer be a problem. Programming will remain very difficult, because once we have freed ourselves from the circumstantial cumbersomeness, we will find ourselves free to tackle the problems that are now well beyond our programming capacity.

You can quarrel with my sixth argument, for it is not so easy to collect experimental evidence for its support, a fact that will not prevent me from believing in its validity. Up till now I have not mentioned the word “hierarchy”, but I think that it is fair to say that this is a key concept for all systems embodying a nicely factored solution. I could even go one step further and make an article of faith out of it, viz. that the only problems we can really solve in a satisfactory manner are those that finally admit a nicely factored solution. At first sight this view of human limitations may strike you as a rather depressing view of our predicament, but I don’t feel it that way, on the contrary! The best way to learn to live with our limitations is to know them. By the time that we are sufficiently modest to try factored solutions only, because the other efforts escape our intellectual grip, we shall do our utmost best to avoid all those interfaces impairing our ability to factor the system in a helpful way. And I cannot but expect that this will repeatedly lead to the discovery that an initially untractable problem can be factored after all. Anyone who has seen how the majority of the troubles of the compiling phase called “code generation” can be tracked down to funny properties of the order code, will know a simple example of the kind of things I have in mind. The wider applicability of nicely factored solutions is my sixth and last argument for the technical feasibility of the revolution that might take place in the current decade.

In principle I leave it to you to decide for yourself how much weight you are going to give to my considerations, knowing only too well that I can force no one else to share my beliefs. As each serious revolution, it will provoke violent opposition and one can ask oneself where to expect the conservative forces trying to counteract such a development. I don’t expect them primarily in big business, not even in the computer business; I expect them rather in the educational institutions that provide today’s training and in those conservative groups of computer users that think their old programs so important that they don’t think it worth-while to rewrite and improve them. In this connection it is sad to observe that on many a university campus the choice of the central computing facility has too often been determined by the demands of a few established but expensive applications with a disregard of the question how many thousands of “small users” that are willing to write their own programs were going to suffer from this choice. Too often, for instance, high-energy physics seems to have blackmailed the scientific community with the price of its remaining experimental equipment. The easiest answer, of course, is a flat denial of the technical feasibility, but I am afraid that you need pretty strong arguments for that. No reassurance, alas, can be obtained from the remark that the intellectual ceiling of today’s average programmer will prevent the revolution from taking place: with others programming so much more effectively, he is liable to be edged out of the picture anyway.

There may also be political impediments. Even if we know how to educate tomorrow’s professional programmer, it is not certain that the society we are living in will allow us to do so. The first effect of teaching a methodology – rather than disseminating knowledge – is that of enhancing the capacities of the already capable, thus magnifying the difference in intelligence. In a society in which the educational system is used as an instrument for the establishment of a homogenized culture, in which the cream is prevented from rising to the top, the education of competent programmers could be politically impalatable.

Let me conclude. Automatic computers have now been with us for a quarter of a century. They have had a great impact on our society in their capacity of tools, but in that capacity their influence will be but a ripple on the surface of our culture, compared with the much more profound influence they will have in their capacity of intellectual challenge without precedent in the cultural history of mankind. Hierarchical systems seem to have the property that something considered as an undivided entity on one level, is considered as a composite object on the next lower level of greater detail; as a result the natural grain of space or time that is applicable at each level decreases by an order of magnitude when we shift our attention from one level to the next lower one. We understand walls in terms of bricks, bricks in terms of crystals, crystals in terms of molecules etc. As a result the number of levels that can be distinguished meaningfully in a hierarchical system is kind of proportional to the logarithm of the ratio between the largest and the smallest grain, and therefore, unless this ratio is very large, we cannot expect many levels. In computer programming our basic building block has an associated time grain of less than a microsecond, but our program may take hours of computation time. I do not know of any other technology covering a ratio of 1010 or more: the computer, by virtue of its fantastic speed, seems to be the first to provide us with an environment where highly hierarchical artefacts are both possible and necessary. This challenge, viz. the confrontation with the programming task, is so unique that this novel experience can teach us a lot about ourselves. It should deepen our understanding of the processes of design and creation, it should give us better control over the task of organizing our thoughts. If it did not do so, to my taste we should not deserve the computer at all!

It has already taught us a few lessons, and the one I have chosen to stress in this talk is the following. We shall do a much better programming job, provided that we approach the task with a full appreciation of its tremendous difficulty, provided that we stick to modest and elegant programming languages, provided that we respect the intrinsic limitations of the human mind and approach the task as Very Humble Programmers.

Writing a basic redis clone in Go from scratch

Introduction

In this tutorial, we will write a basic redis server that is capable of responding to valid RESP requests. In case you are not familiar with what redis is, check out their website. Needless to say, it is an extremely popular caching mechanism that is used everywhere.

However, you might think to yourself that implementing one is a daunting task. I am going to show you that (minus all the critical performance requirements), a redis server is in fact pretty simple. I believe implementing one is a great way to learn about programming. Plus, it also makes you appreciate the engineering challenges of developing a good one.

The full source code is available in this repository.

Let’s begin.

Introduction to REdiS Protocol (RESP)

Before we can interact with a redis application, we need to look at the protocol that it understand. A redis application talks using a protocol called RESP. The protocol is quite simple, the reference documentation fits within a reasonably short HTML page (with examples!).

So, in summary, a RESP object is one of 5 things:

1. A simple string.
2. An error string.
3. An integer value.
4. A bulk string.
5. An array where each element is another RESP object.

A RESP object begins with a flag byte and uses \r\n (will be referred to as CRLF) as a separator or terminator. For example, a RESP simple string looks like +PING\r\n. Where,

1. the first byte (+) indicates the type of data, in this case its a simple string
2. PING is the payload,
3. and ends with a CRLF (\r\n).

Let’s take a look at each object.

Simple Strings

As we have seen above, a simple string:

1. Starts with the + flag byte.
2. Followed by the payload (which are just bytes).
3. Ends with the CRLF.

Now you might be wondering, what if there are \r\n within the payload? For example, something like +PINGTHEN\r\nPONG\r\n where PINGTHEN\r\nPONG is intended to be the payload. Well, simple string simply cannot handle this particular use case. For payloads that may contain \r\n consecutively, one should use bulk strings instead.

Error Strings

Error string is exactly the same as a simple string except that:

1. It begins with the - as the flag byte.
2. It is supposed to be treated as an error message by the client.

Integers

A RESP integer:

1. Starts with : as the flag byte.
2. Followed by the decimal string representation of that number.
3. Ends with the CRLF.

For example, :12354\r\n is a valid integer. The specification does not specify the limit of these integer. However, most implementations will limit it to whatever the host machine can handle. Note that it ONLY supports decimal representation. So something like, :0xffff\r\n is not a valid RESP integer.

Bulk Strings

A bulk string is a little bit more complicated. It,

1. Starts with $ as the flag byte.
2. Followed by a decimal string representation of the length of the string.
3. Followed by the CRLF.
4. Followed by the payload.
5. Then another final CRLF.

Therefore, a ping reponse (like above) will look like $4\r\nPING\r\n instead.

This might seem like a redundant object type since we already have simple strings. Why do we need another string representation? Well, for one, because a bulk string comes with the length of the payload, it can safely have the CRLF as the payload. Another is a matter of performance. Consider what happens if you send/receive a very, very, very long string like a 4MB (33554432 bytes) text of a long novel. If represented as a simple string, the RESP parser would have to iterate over 33554432 bytes looking for the CRLF. However, using bulk strings, they can just check the length of the byte they receive and wait until there are 33554432 bytes.

Arrays

A RESP array is represented simply as multiple RESP objects. It,

1. Begins with the * flag byte.
2. Then the number of elements in the array (also in decimal string representation),
3. Then the CRLF,
4. Then N number of RESP objects.

Visually it looks like this,

*<length-of-array>\r\n<resp><resp><resp>...

For example, a RESP array that consists of 2 simple strings, PING and PONG, looks like,

*2\r\n+PING\r\n+PONG\r\n

It is also possible to have multidimensional RESP arrays. Let’s try to represent something like [["hello","world"], ["goodbye", "world"]] as a RESP array.

1. First, we need the * byte to mark this as a RESP array. So we have *.
2. And we have an array of length 2. Now we have *2.
3. Then we need the CRLF to mark the end of the length. Now we have *2\r\n.
4. Now we have another RESP of length 2…so we just do the same! Now we have *2\r\n*2\r\n.
5. This first inner array contains 2 simple strings hello and world. Now we have *2\r\n*2\r\n+hello\r\n+world\r\n.
6. Then we have another inner array that contains 2 simple strings of goodbye and world. Now we have *2\r\n*2\r\n+hello\r\n+world\r\n*2\r\n+hello\r\n+world\r\n

And that’s it! That’s the entire RESP protocol. We can now try to implement one.

Additional Notes on Bulk String and Array

There’s an additional use for bulk strings and arrays, they can be used to represent the nil (or null) value. Let’s say our Redis server stores an empty set, what should the server return when we query for its member? We could return an error saying “sorry, but it’s empty!” but then the client would have to manually parse and differentiate different kind of error messages. In my opinion, this is bad API design.

So, here’s where the length component of bulk strings and array come in handy. There are 2 kinds of null value in RESP: nil bulk string and nil arrays. They are represented as $-1\r\n and *-1\r\n respectively.

As far as I know, they can be any negative number. I don’t think the particular value have any meaning at all.

Implementation of RESP in Golang

If you look at the description of RESP above, you might notice that we deal a lot with data that ends with CRLF. So it will be very useful to have a helper function that takes some bytes and return the:

1. Bytes up to the CRLF
2. The leftover bytes beginning from the CRLF to the end
3. Whether or not this is valid

// resp.go
func TakeBytesUntilClrf(in []byte) ([]byte, []byte, bool) {
	// If we have empty data, we can't do anything so just return false
	if len(in) == 0 {
		return []byte{}, []byte{}, false
	}

	idx := 0
	// Keep incrementing the index until we have nothing left or we found a CRLF
	for len(in) > idx+1 && !(in[idx] == '\r' && in[idx+1] == '\n') {
		idx++
	}

	// Now check again why did we stop looping, is it because we found a CRLF or because we ran out of data?
	if len(in) > idx+1 && in[idx] == '\r' && in[idx+1] == '\n' {
		// We found the CRLF, return the result
		return in[:idx], in[idx+2:], true
	} else {
		// We didn't find the CRLF, return false
		return in, []byte{}, false
	}
}

Representation of RESP in Golang

First, we need some way to represent RESP as an interface,

// resp.go
type Resp interface {
	// Converts into their byte representation.
	// For example a RESP simple string that contains "hello" will convert into `+hello\r\n`
	AsBytes() []byte

	// static FromBytes([]byte) Resp
}

The AsBytes function here is important because we will be dealing with networking which only understand bytes. Thus, we want a consistent way to convert from our internal Go’s representation into the network stream.

Simple String

// resp.go
type RespSimpleString struct {
	inner string
}

func (rs *RespSimpleString) AsBytes() []byte {
	result := make([]byte, 0, 1+len(rs.inner)+2)
	result = append(result, []byte("+")...)
	result = append(result, []byte(rs.inner)...)
	result = append(result, []byte("\r\n")...)
	return result
}

func RssFromBytes(in []byte) (Resp, []byte) {
	if len(in) == 0 {
		return nil, []byte{}
	} else if in[0] == '+' {
		// For simple strings, we simply get everything from the byte after '+' until CRLF
		str, leftover, ok := TakeBytesUntilClrf(in[1:])

		if !ok {
			return nil, []byte{}
		}

		rs := RespSimpleString{
			inner: string(str),
		}

		return &rs, leftover
	} else {
		return nil, []byte{}
	}
}

Error String

// resp.go
type RespErrorString struct {
	inner string
}

func (rs *RespErrorString) AsBytes() []byte {
	result := make([]byte, 0, 1+len(rs.inner)+2)
	result = append(result, []byte("-")...)
	result = append(result, []byte(rs.inner)...)
	result = append(result, []byte("\r\n")...)
	return result
}

func ResFromBytes(in []byte) (Resp, []byte) {
	if len(in) == 0 {
		return nil, []byte{}
	} else if in[0] == '-' {
		// Similar to error strings
		str, leftover, ok := TakeBytesUntilClrf(in[1:])

		if !ok {
			return nil, []byte{}
		}

		rs := RespErrorString{
			inner: string(str),
		}

		return &rs, leftover
	} else {
		return nil, []byte{}
	}
}

Integer

// resp.go
type RespInteger struct {
	inner int
}

func (rs *RespInteger) AsBytes() []byte {
	v := fmt.Sprintf("%d", rs.inner)
	result := make([]byte, 0, 1+len(v)+2)
	result = append(result, []byte(":")...)
	result = append(result, []byte(v)...)
	result = append(result, []byte("\r\n")...)
	return result
}

func RiFromBytes(in []byte) (Resp, []byte) {
	if len(in) == 0 {
		return nil, []byte{}
	} else if in[0] == ':' {
		// Integer is a little bit interesting, we get everything until CRLF
		str, leftover, ok := TakeBytesUntilClrf(in[1:])

		if !ok {
			return nil, []byte{}
		}

		// Then we try to parse the bytes as a 32-bit integer
		val, err := strconv.ParseInt(string(str), 10, 32)

		if err != nil {
			return nil, []byte{}
		}

		rs := RespInteger{
			inner: int(val),
		}

		return &rs, leftover
	} else {
		return nil, []byte{}
	}
}

Bulk String

// resp.go
type RespBulkString struct {
	inner string
}

func (rs *RespBulkString) AsBytes() []byte {
	a := fmt.Sprintf("%d", len(rs.inner))
	b := []byte(rs.inner)
	result := make([]byte, 0, 1+len(a)+2+len(b)+2)
	result = append(result, []byte("$")...)
	result = append(result, []byte(a)...)
	result = append(result, []byte("\r\n")...)
	result = append(result, b...)
	result = append(result, []byte("\r\n")...)
	return result
}

func RbFromBytes(in []byte) (Resp, []byte) {
	if len(in) == 0 {
		return nil, []byte{}
	} else if in[0] == '$' {
		lenStr, leftover, ok := TakeBytesUntilClrf(in[1:])

		if !ok {
			return nil, []byte{}
		}

		len64, err := strconv.ParseInt(string(lenStr), 10, 32)

		if err != nil {
			return nil, []byte{}
		}

		len32 := int(len64)

		// If its negative, regardless what value it is, we just return nil bulk
		if len32 < 0 {
			return &RespNilBulk{}, leftover
		} else {
			// Since we already have the length of the string that we expected, we don't have to use TakeBytesUntilClrf.
			// Instead, we can simply check at the specified index.
			if len32+1 < len(leftover) && leftover[len32] == '\r' && leftover[len32+1] == '\n' {
				rs := RespBulkString{
					inner: string(leftover[:len64]),
				}

				return &rs, leftover[len64+2:]
			} else {
				return nil, []byte{}
			}
		}
	} else {
		return nil, []byte{}
	}
}

Nil Bulk

// resp.go
type RespNilBulk struct {
}

func (rs *RespNilBulk) AsBytes() []byte {
	return []byte("$-1\r\n")
}

Array

// resp.go
type RespArray struct {
	inner []Resp
}

func (rs *RespArray) AsBytes() []byte {
	blen := fmt.Sprintf("%d", len(rs.inner))
	result := make([]byte, 0, 1+len(blen)+2)
	result = append(result, []byte("*")...)
	result = append(result, []byte(blen)...)
	result = append(result, []byte("\r\n")...)

	for _, r := range rs.inner {
		b := r.AsBytes()
		result = append(result, b...)
	}

	return result
}

func RaFromBytes(in []byte) (Resp, []byte) {
	if len(in) == 0 {
		return nil, []byte{}
	} else if in[0] == '*' {
		lenStr, leftover, ok := TakeBytesUntilClrf(in[1:])

		if !ok {
			return nil, []byte{}
		}

		len64, err := strconv.ParseInt(string(lenStr), 10, 32)

		if err != nil {
			return nil, []byte{}
		}

		len32 := int(len64)

		// We parsed the length of the array, now we march forward
		nextInput := leftover

		if len32 < 0 {
			return &RespNilArray{}, in
		} else {
			replies := make([]Resp, 0, len32)
			for idx := 0; idx < len32 && len(nextInput) != 0; idx++ {
				reply, leftover := TryConvertBytesToResp(nextInput)

				// If any of the elements are bad or we can't make progress, just bail
				if reply == nil {
					return nil, []byte{}
				}

				nextInput = leftover
				replies = append(replies, reply)
			}

			if len(replies) != len32 {
				return nil, []byte{}
			}

			rs := RespArray{
				inner: replies,
			}

			return &rs, nextInput
		}
	} else {
		return nil, []byte{}
	}
}

Nil Array

// resp.go
type RespNilArray struct {
}

func (rs *RespNilArray) AsBytes() []byte {
	return []byte("*-1\r\n")
}

Converting From Bytes to RESP

Putting together all the structs and functions we wrote above, we can implement a function that converts bytes to RESP objects,

// resp.go
func TryConvertBytesToResp(input []byte) (Resp, []byte) {
	// We need at least 1 byte for the first redis type byte
	if len(input) == 0 {
		return nil, []byte{}
	} else if input[0] == '+' { // Simple strings
		resp, leftover := RssFromBytes(input)

		if resp == nil {
			return nil, []byte{}
		}

		return resp, leftover
	} else if input[0] == '-' { // Error strings
		resp, leftover := ResFromBytes(input)

		if resp == nil {
			return nil, []byte{}
		}

		return resp, leftover
	} else if input[0] == ':' { // Integers
		resp, leftover := RiFromBytes(input)

		if resp == nil {
			return nil, []byte{}
		}

		return resp, leftover
	} else if input[0] == '$' { // Bulk strings
		resp, leftover := RbFromBytes(input)

		if resp == nil {
			return nil, []byte{}
		}

		return resp, leftover
	} else if input[0] == '*' { // Arrays
		resp, leftover := RaFromBytes(input)

		if resp == nil {
			return nil, []byte{}
		}

		return resp, leftover
	} else {
		return nil, []byte{}
	}
}

And that should be all there is to our RESP library. The next thing to do is to implement the server and client.

Implementing redis Client

redis already comes with their own standard tool to communicate with it: redis-cli. However, for our use cases (and for fun) we will be implementing our own. One problem with redis-cli is that you cannot test what happens if you send a bad request to redis. redis-cli will automatically sanitize the input for you, which is not what we want for learning purposes.

What we want to do is to manually send a command and then receive a response:

$ redis-client "*1$4\r\nPING\r\n"
+PONG

Therefore, our redis-client is a simple Golang program that:

1. Connects to a port
2. Writes some bytes to that port
3. Read some bytes from that port

Here is the full program plus their explanations,

// cmd/client/client.go
package main

import (
	"fmt"
	"net"
	"os"
	"strings"

	redis "redis_clone"
)

func main() {
	// Check if user passed in an argument
	if len(os.Args) != 2 {
		fmt.Println("This program only accepts 1 argument which is the data to send to port 6379")
		os.Exit(1)
	}

	// Since we will be dealing with bytes, immediately cast the string to []byte
	data := []byte(ConvertCrlf(os.Args[1]))

	// redis server usually uses port 6379 so we will use that too here
	tcpAddr, err := net.ResolveTCPAddr("tcp", "localhost:6379")

	if err != nil {
		fmt.Printf("Failed to resolve TCP address:%s\n", err)
		os.Exit(1)
	}

	tcpConn, err := net.DialTCP("tcp", nil, tcpAddr)

	if err != nil {
		fmt.Printf("Failed to dial TCP:%s\n", err)
		os.Exit(1)
	}

	defer tcpConn.Close()

	WriteData(tcpConn, data)
	resp := ReadData(tcpConn)
	respBytes := resp.AsBytes()
	fmt.Printf("%v\n", EscapeString(string(respBytes)))
}

// WriteData writes data into the tcp connection
func WriteData(tcpConn *net.TCPConn, data []byte) {
	writtenC := 0

	// tcpConn.Write returns the number of bytes that it have written.
	// Therefore, we need to loop and keep trying to write until we have written all the bytes in data.
	// tcpConn.Write only partially writes the data if the buffer on the tcp connection is smaller than our data
	// or if its busy.
	for {
		c, err := tcpConn.Write(data[writtenC:])

		// Connection might be closed while we are writing?
		// In any case, nothing we can do so just abort.
		if err != nil {
			fmt.Printf("Failed to write to connection%s\n", err)
			os.Exit(1)
		}

		writtenC += c

		// We have finally written all the bytes, break from this loop
		if writtenC == len(data) {
			break
		}
	}
}

// WriteData reads data from the tcp connection
func ReadData(tcpConn *net.TCPConn) redis.Resp {
	total := make([]byte, 0)

	// Similar to tcpConn.Write, reading from a tcp connection might not be complete.
	// So we need to keep trying until the bytes that we received can form a complete RESP object.
	for {
		// A buffer of 1024 length means that we will at most 1024 bytes from the connection.
		// This means that if the server is trying to send something bigger than 1024 bytes, we will have to try multiple times.
		// Try changing it to 0 and see what happen (your application will hang).
		buffer := make([]byte, 1024)
		count, err := tcpConn.Read(buffer)

		// If something bad happens, just abort
		if err != nil {
			fmt.Println("Unable to read from connection")
			os.Exit(1)
		}

		total = append(total, buffer[:count]...)

		// Here, we try to parse the bytes we have and see if it can form a RESP
		resp, _ := redis.TryConvertBytesToResp(total)

		// It does! so return from this function with the resp that we got
		if resp != nil {
			return resp
		}
	}
}

// ConvertClrf converts the textual input of \r (which is 2 seperate UTF-8 character) into the actual '\r'
func ConvertCrlf(in string) string {
	in = strings.ReplaceAll(in, "\\r", "\r")
	in = strings.ReplaceAll(in, "\\n", "\n")
	return in
}

// EscapeString does the reverse of ConvertClrf, since printing them as is will cause the output to have a weird format.
func EscapeString(in string) string {
	in = strings.ReplaceAll(in, "\r", "\\r")
	in = strings.ReplaceAll(in, "\n", "\\n")
	return in
}

Implementing redis Server

Before we implement the actual redis server, we can implement a redis echo server that simply returns back whatever RESP input that it’s given. To show that this is a valid redis server, we will try to send data to it using our client and through redis-cli.

The implementation is the following,

// cmd/echo/echo.go
package main

import (
	"fmt"
	"net"
	"os"
	"time"

	redis "redis_clone"
)

func main() {
	// We want to listen to port 6379
	listen, err := net.Listen("tcp", "localhost:6379")

	if err != nil || listen == nil {
		fmt.Printf("Unable to listen to socket:%s\n", err)
		os.Exit(1)
	}

	// 	Loop forever listening for any connections
	for {
		conn, err := listen.Accept()

		if err != nil {
			fmt.Printf("Unable to accept connection:%s\n", err)
			continue
		}

		// Once we get a connection, we start handling it
		conn.SetDeadline(time.Now().Add(1 * time.Second))
		HandleRequest(conn)
		conn.Close()
	}
}

func HandleRequest(conn net.Conn) {
	total := make([]byte, 0)

	for {
		buffer := make([]byte, 1024)
		count, err := conn.Read(buffer)

		if err != nil {
			fmt.Println("Unable to read from socket")
			return
		}

		total = append(total, buffer[:count]...)
		resp, leftover := redis.TryConvertBytesToResp(total)

		if resp != nil {
			total = leftover
			// For now we just reply back the exact same thing
			WriteData(conn, resp.AsBytes())
		}

		if len(leftover) == 0 {
			return
		}
	}
}

func WriteData(conn net.Conn, data []byte) {
	writeC := 0
	for writeC != len(data) {
		read, err := conn.Write(data[writeC:])

		if err != nil {
			fmt.Println("Unable to write to socket")
			return
		}

		writeC += read
	}
}

Now let’s try our server and client. In another shell, run go run cmd/echo/echo.go. Then we can test our redis client by sending valid RESP messages,

$ go run cmd/client/client.go "+PING\r\n"
+PING\r\n
$ go run cmd/client/client.go "*2\r\n+PING\r\n-PING\r\n"
*2\r\n+PING\r\n-PING\r\n
$ go run cmd/client/client.go "-PING\r\n"
-PING\r\n

We can also use the official redis-cli and see that it seems to work!

$ redis-cli -p 6379 set key value
1) "set"
2) "key"
3) "value"
$ redis-cli -p 6379 get key
1) "get"
2) "key"

The reason why we get the output in this form is due to how redis commands work. Basically, most redis command is sent through a RESP array. So when we type,

redis-cli -p 6379 set key value

We are actually sending something like,

*3\r\n$3\r\nset\r\n$3\r\nkey\r\n$5\r\nvalue\r\n

And then, since our current redis server is just echoing back whatever it receives, and this is just how redis-cli displays RESP arrays!

Now what happens if we send invalid RESP objects?

$ go run cmd/client/client.go "!PING\r\n"
Unable to read from connection
exit status 1
$ go run cmd/client/client.go "*2\r\n+PING\r\n"
Unable to read from connection
exit status 1

We can see that our redis server rejects them outright! Let’s try a basic PING command,

$ go run cmd/client/client.go "*1\r\n\$4\r\nPING\r\n"
+PONG\r\n

What about a basic SET command like SET KEY VALUE?

$ go run cmd/client/client.go "*3\r\n\$3\r\nSET\r\n\$3\r\nKEY\r\n\$5\r\nVALUE\r\n"
+OK\r\n

We received the OK from the redis server. And getting them back using GET command like GET KEY?

$ go run cmd/client/client.go "*2\r\n\$3\r\nGET\r\n\$3\r\nKEY\r\n"
$5\r\nVALUE\r\n

And we got our value back!

Implementing GET and SET commands

Now that our server and client can correctly send and receive RESP objects, we now want it to actually do something with it. The 2 most basic redis command is perhaps the GET and SET commands. Using these 2, a client can store data at some key and retrieve them again over the network,

$ redis-cli -p 6379 set key value
OK
$ redis-cli -p 6379 get key
"value"

So we want our redis server to contain some kind of state. Let’s implement one is basically just a dictionary underneath,

// cmd/server/server.go

package main

import (
	"fmt"
	"net"
	"os"
	"strings"
	"time"

	redis "redis_clone"
)

func main() {
	// We want to listen to port 6379
	listen, err := net.Listen("tcp", "localhost:6379")

	if err != nil || listen == nil {
		fmt.Printf("Unable to listen to socket:%s\n", err)
		os.Exit(1)
	}

	state := &State{
		kv: make(map[string]redis.Resp, 0),
	}

	// 	Loop forever listening for any connections
	for {
		conn, err := listen.Accept()

		if err != nil {
			fmt.Printf("Unable to accept connection:%s\n", err)
			continue
		}

		// Once we get a connection, we start handling it
		conn.SetDeadline(time.Now().Add(1 * time.Second))
		HandleRequest(conn, state)
		conn.Close()
	}
}

func HandleRequest(conn net.Conn, state *State) {
	total := make([]byte, 0)

	for {
		buffer := make([]byte, 1024)
		count, err := conn.Read(buffer)

		if err != nil {
			fmt.Println("Unable to read from socket")
			return
		}

		total = append(total, buffer[:count]...)
		resp, leftover := redis.TryConvertBytesToResp(total)

		if resp != nil {
			total = leftover
			// HandleRequest(conn)
			resl := state.HandleResp(resp)
			WriteData(conn, resl.AsBytes())
		}

		if len(leftover) == 0 {
			return
		}
	}
}

func WriteData(conn net.Conn, data []byte) {
	writeC := 0
	for writeC != len(data) {
		read, err := conn.Write(data[writeC:])

		if err != nil {
			fmt.Println("Unable to write to socket")
			return
		}

		writeC += read
	}
}

type State struct {
	// Dictionary of map to RESP
	kv map[string]redis.Resp
}

// Get returns the RESP at key if it exists, otherwise returns nil
func (s *State) Get(key string) redis.Resp {
	v, e := s.kv[key]
	if !e {
		return nil
	} else {
		return v
	}
}

// Set simply maps key to some value. Overwrites the old value
func (s *State) Set(key string, value redis.Resp) {
	s.kv[key] = value
}

// HandleResp takes the RESP, processes it and returns RESP response
func (s *State) HandleResp(resp redis.Resp) redis.Resp {
	// The default error. redis have various kind of error messages
	errResult := &redis.RespErrorString{
		Inner: "please provide a valid redis command",
	}

	rarr := make([]redis.Resp, 0)

	// Make sure that the RESP we received is a RESP array
	if arr, ok := resp.(*redis.RespArray); ok {
		rarr = arr.Inner
	}

	// If its empty, return error (or if the above if failed)
	if len(rarr) == 0 {
		return errResult
	}

	sarr := make([]*redis.RespBulkString, 0, len(rarr))

	// Each RESP in the array must be a bulk string
	for _, r := range rarr {
		if r, ok := r.(*redis.RespBulkString); ok {
			sarr = append(sarr, r)
		} else {
			return errResult
		}
	}

	if strings.ToLower(sarr[0].Inner) == "get" { // GET command
		if len(sarr) != 2 { // GET command requires exactly 2 arguments i.e GET <key>
			return errResult
		} else {
			v := s.Get(sarr[1].Inner)

			if v == nil {
				return &redis.RespNilArray{}
			} else {
				return v
			}
		}
	} else if strings.ToLower(sarr[0].Inner) == "set" { // SET command
		if len(sarr) != 3 { // SET command requires exactly 2 arguments i.e SET <key> <value>
			return errResult
		} else {
			s.Set(sarr[1].Inner, sarr[2])
			return &redis.RespSimpleString{Inner: "OK"}
		}
	} else { // Reject every other commands
		return &redis.RespErrorString{Inner: fmt.Sprintf("command %s is not yet supported", sarr[0].Inner)}
	}
}

Now let’s try using our redis server and client, and see if it works! Start the server and execute,

$ go run cmd/client/client.go "*3\r\n\$3\r\nSET\r\n\$3\r\nKEY\r\n\$5\r\nhello\r\n"
+OK\r\n
$ go run cmd/client/client.go "*2\r\n\$3\r\nGET\r\n\$3\r\nKEY\r\n"
$5\r\nhello\r\n

Awesome! You’ve now implement a basic redis clone! Well…now what? Here’s how you can make this implementation better:

1. Implement all the commands here…. Now that’s a lot of commands. But this should give you a foundation to implement many of them (try to implement LPUSH, LPOP).
2. You can also try to make this multithreaded using a RWLock!
3. Support blocking commands?
4. Persistency by writing to disk?

Programming with Context (draft)

Disclaimer

I have no formal traning in functional programming so if I have any mistakes here please feel free to correct and educate me!

Introduction

In programming, we have to deal with a lot of complexities. In my experience, there are 2 different kinds of complexities:

1. Business complexity. This is the complexity of the requirement that have been tasked to us developers. For example, a requirement that says “User can go to our website to browse our catalogue”. This is inherent.

2. Programming complexity. This is the complexity that comes from the implementation of said requirements. For example, a requirement to develop a website makes no mention of HTTP but as implementators we have to be aware of it.

One thing about complexity is that they will always exist. We may be able to reduce the latter kind of complexity (which is why certain abstraction fails and some do not). However, the former kind of complexity will always be there — it literally has to do that thing that is required of it.

I think one of the biggest cause of complexity is states. The problem is that our monkey brain is extremely bad at managing them (Or maybe it’s just me). There some tools to help us narrow down the set of possible issues/states that we have to deal with. Types is one such tool and it is possibly the most popular and reliable static analysis tool.

The only thing that we can do is to abstract it away the complexity. Then we just pray that the abstraction doesn’t break in a really awful manner down the line. At the end of the day, someone gotta pay for the complexity.

We Begin with Dynamic Types

Dynamic languages will “allow” us to make our computers do “anything we like”. I say in quotes because this is actually not true. The reason is that we most definitely do not want our computers to do “anything we like”. It is very hard for us human to keep a consistent flow of thought. So if a machine is willing to do anything I tell it to, well, most of time, its going to be doing the wrong thing. It’s very hard for us human to be precise with our desires and even worse to communicate it. We make a lot of assumptions that things work the way we think it does, not the way it actually does work.

My gripe with untyped languge is that it allows you to do even the stupidest mistake. It causes the set of possible programs (mostly wrong programs) to be extremely large. For example, consider the following JavaScript code,

let a = "a";
let b = false;
let c = a + b;

Does this make any sense? This piece of code is most definitely meaningless.

Another example is C’s printf.

const char *pointer = "hello world";
printf("%%p => %p\n", pointer);
printf("%%s => %s\n", pointer);
printf("%%d => %d\n", pointer);

Compiling and executing the binary gets us,

hbina@akarin:~/git/hbina.github.io$ clang ./static/programming_with_context/example_printf_program.c && ./a.out
./static/programming_with_context/example_printf_program.c:8:27: warning: format specifies type 'int' but the argument has type 'const char *' [-Wformat]
    printf("%%d => %d\n", pointer);
                   ~~     ^~~~~~~
                   %s
1 warning generated.
%p => 0x402004
%s => hello world
%d => 4202500

The third usage of printf is most definitely wrong and meaningless. Fortunately clang does warn us about this. Regardless, it still pains me that a computer is allowing me to make such mistakes. The compiler is telling me “This is literally nonsense, it will not do what you think it does” and yet it still happily compiles it for me. Why? printf is mostly stateless, so it’s “not hard” to get it right by looking at individual uses. However, imagine dealing with C’s tagged union.

I think dynamic languages is the perfect tool for writing small scripts. However, the moment you want the program to span for more than 2 files or when you want to refactor/enhance the functionality; you will start having issues or implicit conversions?

In my experience, what ended up happening is some sort of ad-hoc type-checking anyways.

1. Perform a manual check/verification. You do this by going through all the possible states of the program to convince yourself that the function does what you think it does. Needless to say this is insanely hard to do. Most of the time, its just assumed that something works and hope for the best. When something does go bad, then we go take a look and fix it. This is obviously not ideal.

2. Another way you can narrow down the states (and to verify your assumptions) is to perform some kind of check. For example, we can do a type-guard IN JavaScript. This way we check that any given value is what you think it is.

if (typeof value === "string") {
  // Great! It's a string
} else {
  // Do something else...
}

3. Write lots of test cases. Loads of them.

Enter Types

To me, type is just a way to constraint the possible states that a variable can take. If I annotated a variable as a number, then I don’t have to think about it being anything other than a number. This alone removes a huge cognitive load from anyone attempting to understand the behavior of the program.

Additionally, types also provides you hint about the implementation of any given function. For example, a function that takes 2 numbers and returning a number will probably do some sort of calculations on them. Given a function like,

const HighestBidder = (bids: [String, number][]) -> [String, number] => {...}

I can already expect what the function does. If I want to be extra sure and really understand the implementation, the function name and the types will guide me through it.

However, types alone is not powerful enough to constraint the possible states and especially unique/bad ones.

Function + Values = Consequence

Eventhough we have narrowed down a variable to just one type, some types can take on many different values with varying meanings. For example, if we have a type number it includes all the possible 64-bit floating points.

Therefore, we can think of a type as having a set of values. And within this set, the values are divided into what we consider as “normal values” and some “problematic values”.

But, these values existing by itself isn’t the problem. I can declare any values I want, nullptr, NaN, -Inf or whatever and if it’s just sitting there not interacting with anything, it’s by definition is doing nothing except maybe take up stack space.

However, applying functions to values have consequences.

The problem arises when we interact with these values using functions. When we write a function, we are attempting to express some idea of computation. The arguments type indicate what possible values the function can take and the result type indicate what possible values it will spit out.

Let’s consider an addition function,

const addition = (a: number, b: number): number => {
  return a + b;
};

But we have a problem here. We are interacting with the whole set of possible values of number. And some of them are not as nice to work as we have discovered. What does NaN + 3 means? What about 3 + -Inf? All of these are values that we really don’t care.

I mean sure, someone will come up and say they know every possible interactions by hand. But it’s still silly to worry about that every single time I want to do anything meaningful. I am just trying to write an addition function here!

This is where the second kind of complexity that I mentioned before comes sneaking in. Most requirements most definitely do not include a specification for how to deal with NaN or floating point errors. Yet it is something that we have to consider.

One easy way to go around this is to,

const addition = (a: number, b: number): number | undefined => {
  if (a === NaN || a === Infinity || a === -Infinity /* and the rest */) {
    return undefined;
  } else {
    return a + b;
  }
};

At this point, you are pretty happy about the safety of this function. We are sure that any time we call this function, the result is a number. We are confident that the computation we are performing here are what we consider as “addition”.

But then, a new requirement comes in and we need to implement a substraction function. Fine, we can just copy and paste the implementation before and replace + with -.

const substraction = (a: number, b: number): number | undefined => {
  if (a === NaN || a === Infinity || a === -Infinity /* and the rest */) {
    return undefined;
  } else {
    return a - b;
  }
};

Then multiple new requirement comes in, multiplication, division, factorial… We will need to do this every single time. Even worse than that, we can clearly see that they are all awfully similar. Notice that the only thing that “matters” here is the binary operator. Just that 1 character.

Surely there’s some way to abstract/pull out these common checks into a class.

Don’t Repeat Yourself

Let’s digress a bit and talk about function that accepts pointers as its argument. A pointer can take on many values, so the function itself has to deal with the states. From the function’s point of view, the pointer could take on any values so it has to check if its NULL or not. This is bad because:

1. We need to always do it.
2. We need to always do it right.
3. We need to pinky promise that we will do it all the time.

It is simply nigh impossible to know when/where any given pointers are valid throughout its existence. From my experience, what ended up happening is that people create macros that will check for null pointers and panic otherwise. This macros would then be plastered around everywhere at the beginning of every function that uses pointers. Some of them are manually removed if the author is convinced enough that its not needed.

Can we do better? Aren’t we taugh to pull out common abstraction into a function and reuse that instead?

Hide Behind the Container

It turns that there is a way to deal with all this repetitiveness. If you have ever taken a class in computer science or programming, you will most definitely have come across classes (or containers). The strategy is that we hide the values behind a container and the only way you can interact with the contained value is by passing it a function. Let’s call this a MaybeNumber.

class MaybeNumber {
  value: number;
  constructor(v: number) {
    this.value = v;
  }
  bind(f: (t: number) => MaybeNumber): MaybeNumber {
    if (
      this.value === NaN ||
      this.value === Infinity ||
      this.value === -Infinity /* and the rest */
    ) {
      return this;
    } else {
      return f(this.value);
    }
  }
}

Let’s go through the class slowly. First, we note that the class contains 1 member variable, value which contains the number that we want to hide. Then we have a simply constructor to create this container from a pure number. Lastly, we have the bind function that takes a number and returns the MaybeNumber. You might be thinking, why does it return a MaybeNumber and not a Inumber?

This is a deliberate choice because we want to force the user of this value to always deal with this class and it enables nice composition ala Promises.

First, lets deal with the fact that the signature requires us to return a MaybeNumber. Does this mean that we need to reimplement all the functions that didn’t use MaybeNumber into one? Fear not, for we can lift these functions.

We can simply “lift” any given function using the unit function. lift takes any function from T -> number into T -> MaybeNumber

type BindArg = (a: number) => number;
type BindResult = (a: number) => MaybeNumber;
function lift(f: BindArg): BindResult {
  return (a) => unit(f(a));
}

And so if we have functions like,

const times2 = (v: number) => v * 2;
const adds2 = (v: number) => v + 2;

We can simply lifts them and use them like so,

const times2M = lift(times2);
const adds2M = lift(adds2);

const b = unit(10);
const b2 = b.bind(times2M);
const b2M = b2.bind(times2M).bind(adds2M).bind(times2M).bind(adds2M);

Enter Monads

In my (humble) opinion, monads is just a way to express computations as types. Computations produces result and instead of actually using the result, we just provide a function f to the type that performs the computation itself.

A monad is not any specific type, it is more like an instance. In Haskell, Monad is a typeclass defined as,

class  Monad m  where
    -- The () indicates that this is an infix operator, so it closely resembles a class having a method.
    (>>=)  :: m a -> (a -> m b) -> m b
    return :: a -> m a

TypeScript cannot express this concept but let’s pretend we have one that can. In this hypothetical TypeScript, it could be written like this,

export interface Monad<T> {
  // also called unit
  static return(t: T): Monad<T>;
  // The monad's >== function is also called bind
  bind(f: (t: T) => Monad<R>): Monad<R>;
}

And any class M<T> that implements Monad<T> (satisfying the monadic laws which we will not cover here) is a monad. What is so powerful about this? Well, it enables you to abstract one computation from another.

Let’s look at some example.

Maybe Monad

The maybe monad is used to represent a “computation” or “behavior” where a value may or may not exist. Let’s implement it in our imaginary TypeScript,

// An enum to express existence of a value

type Option<T> =
  | {
      kind: "Some";
      value: T;
    }
  | {
      kind: "Nothing";
    };

// Helper functions

const createSome = <T>(t: T) => {
    kind : "Some",
    value : t
}

const createNone = <T>() => {
    kind : "Nothing",
}

class Maybe<Option<T>> implements Monad<Option<T>> {
    t : Option<T>
    static return(t: Option<T>) => new Maybe(t)
    bind(f: (t: T) => Maybe<Option<R>>): Maybe<Option<R>> {
        if (this.t.kind === "Nothing") {
            return createNone();
        } else {
            return f(this.t.value);
        }
    }
}

And using it looks like,

// Here we implement the behavior that division by 0 is bad.
const safeAddition = (a: PositiveInteger, b: PositiveInteger) => {
  if (MAX_POSITIVE_INT - b > a) {
      // Overflow, returns none
      return Maybe::return(createNone());
  } else {
      // Addition does not overflow
      return Maybe::return(createSome(a / b));
  }
}

// Here we implement a funct
const multiplication = (
  a: Maybe<Option<T>>,
  b: Maybe<Option<T>>
): Maybe<Option<T>> => {
  return a.bind((l) =>
    b.bind((r) => (l === 1 ?
      r : r === 1 ? l : safeAddition(l, l).bind((v) =>
        multiplication(
          Maybe::return(createSome(v)),
          Maybe::return(createSome(r - 1)))))
    ));
};

I am going to be honest, it doesn’t look as great. This is mostly because you need to jump through lot of syntax to do anything meaningful. This is why Haskell does away with almost all syntax to function application (the only thing that matters really is the position). This makes Haskell’s implementation of this concept a lot more compact.

One thing to notice is that we didn’t have to perform any check on the behavior that “division by 0 is bad” anywhere in multiplication. The error handling is done automatically and we only concern ourselves with the details to implement multiplication.

Implement allCombinations

List Monad

State Monad

Some Examples of List Homomorphism (draft)

Introduction

List homomorphism is a powerful concept that allows us to turn a computation into an embarrasingly parallel problem.

Code

// Basically need to satisfy this constraint
// f(list_one + list_two) === homo_list(f(list_one), f(list_two));
// The function 'f' is a list homomorphism if there exists a function `homo_list` that satisfy the above equation

type IntermediateState = {
  mss: number; // the actual MSS of a given array
  mss_from_left: number; // the MSS if we force subarray to include the first element
  mss_from_right: number; // the MSS if we force the subarray to include the last element
  total_sum: number; // simply the total sum of the array
};

// Transforming a single element into intermediate state is trivial
const f = (input: number): IntermediateState => {
  return {
    mss: input,
    mss_from_left: input,
    mss_from_right: input,
    total_sum: input,
  };
};

const dot = (
  lhs: IntermediateState,
  rhs: IntermediateState
): IntermediateState => {
  return {
    mss: Math.max(lhs.mss, rhs.mss, lhs.mss_from_right + rhs.mss_from_left),
    mss_from_left: Math.max(
      lhs.mss_from_left,
      lhs.total_sum + rhs.mss_from_left
    ),
    mss_from_right: Math.max(
      rhs.mss_from_right,
      lhs.mss_from_right + rhs.total_sum
    ),
    total_sum: lhs.total_sum + rhs.total_sum,
  };
};

const h = (input: Array<number>): IntermediateState => {
  if (input.length === 0) {
    return {
      mss: 0,
      mss_from_left: 0,
      mss_from_right: 0,
      total_sum: 0,
    };
  } else {
    return input.map((x) => f(x)).reduce((acc, curr) => dot(acc, curr));
  }
};

althttpd: From Top to Bottom

Before Venturing Further…

It’s nice to have the codebase open as reference. Each chapters will cover mostly 1 function and I wouldn’t go through every single bit. Just the parts that I find interesting. I will cover the global variables as we go through.

Sometimes there will be backtracking and jumping ahead of the source code as I see necessary.

Please read about the design philosophy here.

Check out the source code here.

With that out of the way…Let’s go!

Functions

Escape

This function accepts a C-string and returns another C-string with every double-quote doubled.

/*
** Double any double-quote characters in a string.
*/
static char *Escape(char *z){
  size_t i, j;
  size_t n;
  char c;
  char *zOut;
  for(i=0; (c=z[i])!=0 && c!='"'; i++){}
  if( c==0 ) return z;
  n = 1;
  for(i++; (c=z[i])!=0; i++){ if( c=='"' ) n++; }
  zOut = malloc( i+n+1 );
  if( zOut==0 ) return "";
  for(i=j=0; (c=z[i])!=0; i++){
    zOut[j++] = c;
    if( c=='"' ) zOut[j++] = c;
  }
  zOut[j] = 0;
  return zOut;
}

First we have the variable declarations at the top of the function… So its one of those code. sigh. Second, we can see that there is 0 calls to strlen. We perform a single pass on the original strings and a second one on the output string. Pretty cool.

Notice that the caller of this function have no real way to know if allocations have been made. However, from the design philosophy,

Because each althttpd process only needs to service a single connection, althttpd is single threaded.
Furthermore, each process only lives for the duration of a single connection, which means that althttpd does not need to worry too much about memory leaks.
These design factors help keep the althttpd source code simple, which facilitates security auditing and analysis.

So this seems like a conscious desicion. If you grep for this function, its only being used in one place.

bash-3.2$ rg 'Escape' ./content/althttpd.c
387:static char *Escape(char *z){
481:        zDate, zRemoteAddr, zHttp, Escape(zHttpHost), Escape(zScript),
482:        Escape(zReferer), zReplyStatus, nIn, nOut,
488:        nRequest, Escape(zAgent), Escape(zRM),

Notice that they don’t even assign the pointer returned by this function! I will let you judge whether this is good idea or not.

tvms

This function is pretty much self explanatory. It is what is… I guess I can start reading about the time utility in libc.

/*
** Convert a struct timeval into an integer number of microseconds
*/
static long long int tvms(struct timeval *p){
  return ((long long int)p->tv_sec)*1000000 + (long long int)p->tv_usec;
}

References

1. gettimeofday

MakeLogEntry

This function is quite big. Lets dig in.

/*
** Make an entry in the log file.  If the HTTP connection should be
** closed, then terminate this process.  Otherwise return.
*/
static void MakeLogEntry(int exitCode, int lineNum){
  FILE *log;
  if( zTmpNam ){
    unlink(zTmpNam);
  }
  if( zLogFile && !omitLog ){
    struct timeval now;
    struct tm *pTm;
    struct rusage self, children;
    int waitStatus;
    char *zRM = zRemoteUser ? zRemoteUser : "";
    char *zFilename;
    size_t sz;
    char zDate[200];
    char zExpLogFile[500];

    if( zScript==0 ) zScript = "";
    if( zRealScript==0 ) zRealScript = "";
    if( zRemoteAddr==0 ) zRemoteAddr = "";
    if( zHttpHost==0 ) zHttpHost = "";
    if( zReferer==0 ) zReferer = "";
    if( zAgent==0 ) zAgent = "";
    gettimeofday(&now, 0);
    pTm = localtime(&now.tv_sec);
    strftime(zDate, sizeof(zDate), "%Y-%m-%d %H:%M:%S", pTm);
    sz = strftime(zExpLogFile, sizeof(zExpLogFile), zLogFile, pTm);
    if( sz>0 && sz<sizeof(zExpLogFile)-2 ){
      zFilename = zExpLogFile;
    }else{
      zFilename = zLogFile;
    }
    waitpid(-1, &waitStatus, WNOHANG);
    getrusage(RUSAGE_SELF, &self);
    getrusage(RUSAGE_CHILDREN, &children);
    if( (log = fopen(zFilename,"a"))!=0 ){
#ifdef COMBINED_LOG_FORMAT
      strftime(zDate, sizeof(zDate), "%d/%b/%Y:%H:%M:%S %Z", pTm);
      fprintf(log, "%s - - [%s] \"%s %s %s\" %s %d \"%s\" \"%s\"\n",
              zRemoteAddr, zDate, zMethod, zScript, zProtocol,
              zReplyStatus, nOut, zReferer, zAgent);
#else
      strftime(zDate, sizeof(zDate), "%Y-%m-%d %H:%M:%S", pTm);
      /* Log record files:
      **  (1) Date and time
      **  (2) IP address
      **  (3) URL being accessed
      **  (4) Referer
      **  (5) Reply status
      **  (6) Bytes received
      **  (7) Bytes sent
      **  (8) Self user time
      **  (9) Self system time
      ** (10) Children user time
      ** (11) Children system time
      ** (12) Total wall-clock time
      ** (13) Request number for same TCP/IP connection
      ** (14) User agent
      ** (15) Remote user
      ** (16) Bytes of URL that correspond to the SCRIPT_NAME
      ** (17) Line number in source file
      */
      fprintf(log,
        "%s,%s,\"%s://%s%s\",\"%s\","
           "%s,%d,%d,%lld,%lld,%lld,%lld,%lld,%d,\"%s\",\"%s\",%d,%d\n",
        zDate, zRemoteAddr, zHttp, Escape(zHttpHost), Escape(zScript),
        Escape(zReferer), zReplyStatus, nIn, nOut,
        tvms(&self.ru_utime) - tvms(&priorSelf.ru_utime),
        tvms(&self.ru_stime) - tvms(&priorSelf.ru_stime),
        tvms(&children.ru_utime) - tvms(&priorChild.ru_utime),
        tvms(&children.ru_stime) - tvms(&priorChild.ru_stime),
        tvms(&now) - tvms(&beginTime),
        nRequest, Escape(zAgent), Escape(zRM),
        (int)(strlen(zHttp)+strlen(zHttpHost)+strlen(zRealScript)+3),
        lineNum
      );
      priorSelf = self;
      priorChild = children;
#endif
      fclose(log);
      nIn = nOut = 0;
    }
  }
  if( closeConnection ){
    exit(exitCode);
  }
  statusSent = 0;
}

Grepping for the usage of this function, we get:

bash-3.2$ rg 'MakeLogEntry' ./content/althttpd/althttpd.c
417:static void MakeLogEntry(int exitCode, int lineNum){
514:    MakeLogEntry(1,100);  /* LOG: Malloc() failed */
708:  MakeLogEntry(0, lineno);
723:  MakeLogEntry(0, lineno);
741:  MakeLogEntry(0, 110);  /* LOG: Not authorized */
756:  MakeLogEntry(0, 120);  /* LOG: CGI Error */
773:      MakeLogEntry(0, 130);  /* LOG: Timeout */
789:  MakeLogEntry(0, 140);  /* LOG: CGI script is writable */
809:  MakeLogEntry(0, linenum);
840:    MakeLogEntry(0, lineno);
1323:    MakeLogEntry(0, 470);  /* LOG: ETag Cache Hit */
1347:    MakeLogEntry(0, 2); /* LOG: Normal HEAD reply */
1684:    MakeLogEntry(0, 200); /* LOG: bad protocol in HTTP header */
1709:    MakeLogEntry(0, 220); /* LOG: Unknown request method */
1908:      MakeLogEntry(0, 270); /* LOG: Request too large */
1926:      MakeLogEntry(0, 290); /* LOG: cannot create temp file for POST */
2219:  MakeLogEntry(0, 0);  /* LOG: Normal reply */

It seems that most (if not all) of these usages are related to logging before killing the connection process. It accepts the exitCode that the process should return, and a lineNum.

The second parameter is supposed to indicate the approximate line number this function is called. Probably for debugging purposes.

The first thing we do in the function is to unlink a file.

if( zTmpNam ){
  unlink(zTmpNam);
}

Grepping for zTmpNam yields,

bash-3.2$ rg 'zTmpNam' ./content/althttpd/althttpd.c
290:static char *zTmpNam = 0;        /* Name of a temporary file */
291:static char zTmpNamBuf[500];     /* Space to hold the temporary filename */
419:  if( zTmpNam ){
420:    unlink(zTmpNam);
1310:  if( zTmpNam ) unlink(zTmpNam);
1610:   && (in = fopen(zTmpNam,"r"))!=0 ){
1912:    sprintf(zTmpNamBuf, "/tmp/-post-data-XXXXXX");
1913:    zTmpNam = zTmpNamBuf;
1914:    if( mkstemp(zTmpNam)<0 ){
1918:    out = fopen(zTmpNam,"wb");
1924:        "Could not open \"%s\" for writing\n", zTmpNam
2155:      open(zTmpNam, O_RDONLY);

So zTmpNam is basically a temporary file created from mkstemp. Then, if zLogFile is provided and omitLog is false, we start some logging procedures. This block of statements here:

if( zScript==0 ) zScript = "";
if( zRealScript==0 ) zRealScript = "";
if( zRemoteAddr==0 ) zRemoteAddr = "";
if( zHttpHost==0 ) zHttpHost = "";
if( zReferer==0 ) zReferer = "";
if( zAgent==0 ) zAgent = "";
gettimeofday(&now, 0);
pTm = localtime(&now.tv_sec);
strftime(zDate, sizeof(zDate), "%Y-%m-%d %H:%M:%S", pTm);
sz = strftime(zExpLogFile, sizeof(zExpLogFile), zLogFile, pTm);
if( sz>0 && sz<sizeof(zExpLogFile)-2 ){
  zFilename = zExpLogFile;
}else{
  zFilename = zLogFile;
}
waitpid(-1, &waitStatus, WNOHANG);
getrusage(RUSAGE_SELF, &self);
getrusage(RUSAGE_CHILDREN, &children);

Is basically to get the current state of the process that will be logged. The other interesting part of the program is in the logging itself.

fprintf(log,
  "%s,%s,\"%s://%s%s\",\"%s\","
      "%s,%d,%d,%lld,%lld,%lld,%lld,%lld,%d,\"%s\",\"%s\",%d,%d\n",
  zDate, zRemoteAddr, zHttp, Escape(zHttpHost), Escape(zScript),
  Escape(zReferer), zReplyStatus, nIn, nOut,
  tvms(&self.ru_utime) - tvms(&priorSelf.ru_utime),
  tvms(&self.ru_stime) - tvms(&priorSelf.ru_stime),
  tvms(&children.ru_utime) - tvms(&priorChild.ru_utime),
  tvms(&children.ru_stime) - tvms(&priorChild.ru_stime),
  tvms(&now) - tvms(&beginTime),
  nRequest, Escape(zAgent), Escape(zRM),
  (int)(strlen(zHttp)+strlen(zHttpHost)+strlen(zRealScript)+3),
  lineNum
);

Remember the function Escape? This is the ony place where its being used and the pointer returned isn’t assigned to anything. So we can’t possibly free any of the allocations made.

However, just after this, we have:

if( closeConnection ){
  exit(exitCode);
}

Additionally based on the grepped usage of this function above. We saw that most of the time, they immediately exit process. So all those “memory leaks” previously (AFAIK) will be inconsequential.

References

1. mkstemp
2. unlink
3. gettimeofday
4. localtime
5. strftime
6. exit

SafeMalloc

This function is a wrapper for malloc where it will exit the process if it fails. I think the function itself is self-explanatory. Nothing really interesting is happening here.

/*
** Allocate memory safely
*/
static char *SafeMalloc( size_t size ){
  char *p;

  p = (char*)malloc(size);
  if( p==0 ){
    strcpy(zReplyStatus, "998");
    MakeLogEntry(1,100);  /* LOG: Malloc() failed */
    exit(1);
  }
  return p;
}

References

1. strcpy
2. exit

SetEnv

This function sets the environment variables.

/*
** Set the value of environment variable zVar to zValue.
*/
static void SetEnv(const char *zVar, const char *zValue){
  char *z;
  size_t len;
  if( zValue==0 ) zValue="";
  /* Disable an attempted bashdoor attack */
  if( strncmp(zValue,"() {",4)==0 ) zValue = "";
  len = strlen(zVar) + strlen(zValue) + 2;
  z = SafeMalloc(len);
  sprintf(z,"%s=%s",zVar,zValue);
  putenv(z);
}

There’s some issues with this function.

1. It does not free the memory held by z.
2. I think their implementation to prevent bashdoor (Whatever that is) attack is somewhat naive? I am pretty sure one can bypass this check by simply prepending whitespaces.
3. It does not check if zVar is a NULL. Granted, if you grep for its usage, you will find that it only uses statically allocated memory.

bash-3.2$ rg 'SetEnv' ./static/althttpd/althttpd.c
523:static void SetEnv(const char *zVar, const char *zValue){
2136:        SetEnv(cgienv[i].zEnvName,*cgienv[i].pzEnvValue);

The cgienv here is an array of CGI environment variables defined at the top of the file. These are all statically allocated, so no problem here. However, the performance could have been improved in this case by passing the length of the arrays into the function. I am not sure if compilers are smart enough (or allowed to) optimized the call to strlen by evaluating the size at compile time.

References

1. setenv

GetFirstElement

This function is basically strtok with a slightly different interface. It accepts:

1. zInput, the C-string to be tokenized.
2. zLeftOver, an output pointer that points to to the leftover C-string.

And returns the parsed token.

/*
** Remove the first space-delimited token from a string and return
** a pointer to it.  Add a NULL to the string to terminate the token.
** Make *zLeftOver point to the start of the next token.
*/
static char *GetFirstElement(char *zInput, char **zLeftOver){
  char *zResult = 0;
  if( zInput==0 ){
    if( zLeftOver ) *zLeftOver = 0;
    return 0;
  }
  while( isspace(*(unsigned char*)zInput) ){ zInput++; }
  zResult = zInput;
  while( *zInput && !isspace(*(unsigned char*)zInput) ){ zInput++; }
  if( *zInput ){
    *zInput = 0;
    zInput++;
    while( isspace(*(unsigned char*)zInput) ){ zInput++; }
  }
  if( zLeftOver ){ *zLeftOver = zInput; }
  return zResult;
}

Visualization

h | e | l | l | o | NULL | | | | w | o | r | l | d |
|--------------------|-----------|------------------...
a                    b           c

Mark the beginning of the pointer as a. The implementation is similar to strtok, it loops through the string until it encounters a whitespace. It then loop again until it encounters non-whitespace and mark this as c. Assign c to the zLeftOver pointer and returns a as the result. To get the next token, pass zLeftOver token in zInput.

Note that because this function will litter the original C-string with NULLs, it will become unusable by strlen. You can audit whether or not the author ever uses strlen on the C-strings that have been passed into this function.

Obviously the above algorithm needs to take into consideration the NULL value that marks the end of a C-string.

References

1. strtok

StrDup

This function basically copies a C-string. Nothing special.

/*
** Make a copy of a string into memory obtained from malloc.
*/
static char *StrDup(const char *zSrc){
  char *zDest;
  size_t size;

  if( zSrc==0 ) return 0;
  size = strlen(zSrc) + 1;
  zDest = (char*)SafeMalloc( size );
  strcpy(zDest,zSrc);
  return zDest;
}

However, I personally think that using memcpy is better here because we already calculated the string length here.

References

1. strlen
2. strcpy
3. memcpy

StrAppend

This function basically takes 3 C-string and concatenates them.

static char *StrAppend(char *zPrior, const char *zSep, const char *zSrc){
  char *zDest;
  size_t size;
  size_t n0, n1, n2;

  if( zSrc==0 ) return 0;
  if( zPrior==0 ) return StrDup(zSrc);
  n0 = strlen(zPrior);
  n1 = strlen(zSep);
  n2 = strlen(zSrc);
  size = n0+n1+n2+1;
  zDest = (char*)SafeMalloc( size );
  memcpy(zDest, zPrior, n0);
  free(zPrior);
  memcpy(&zDest[n0],zSep,n1);
  memcpy(&zDest[n0+n1],zSrc,n2+1);
  return zDest;
}

Visualization

Take 3 C-strings:

1. zPrior as hello.
2. zSep as my.
3. zSrc as world.

After this function, we will get something like,

h | e | l | l | o |  | m | y | | w | o | r | l | d | NULL |
|-----------------|------------|---------------------------...
<     zPrior      ><    zSep   ><         zSrc          >

And the original zPrior is freed, for whatever reason.

References

1. strlen.
2. memcpy.
3. free.

CompareEtags

This function compares 2 C-strings and returns 0 if they differ and non-zero otherwise.

/*
** Compare two ETag values. Return 0 if they match and non-zero if they differ.
**
** The one on the left might be a NULL pointer and it might be quoted.
*/
static int CompareEtags(const char *zA, const char *zB){
  if( zA==0 ) return 1;
  if( zA[0]=='"' ){
    int lenB = (int)strlen(zB);
    if( strncmp(zA+1, zB, lenB)==0 && zA[lenB+1]=='"' ) return 0;
  }
  return strcmp(zA, zB);
}

Notice that the function never checks the length of lenB, so it might seem problematic that we perform an arbitrary index zA[lenB + 1] here. I think this is totally safe because the check is implicitly done by the fact that:

1. All C-strings ends with a NULL. This is kind of hard to verify.
2. In the if logic, we will only evaluate the right-hand side iff the call to strncmphere succeed. This means that zA is at least as long as lenB. Coupled with the fact that its a C-string, we can be sure that there’s at least another character beyond that, namely the NULL character.

References

1. strlen.
2. strncmp.

RemoveNewline

This function replaces newline (\n or \r) with NULL.

/*
** Break a line at the first \n or \r character seen.
*/
static void RemoveNewline(char *z){
  if( z==0 ) return;
  while( *z && *z!='\n' && *z!='\r' ){ z++; }
  *z = 0;
}

This is probably done so that C-string appears as if it ends there. But this is highly problematic because it does not return the z. The caller have no way of knowing where that NULL was assigned. Even if they loop through again and find that NULL, how do they know that this NULL is the one that we assigned or this is the actual the end of the C-string?

In my opinion, it should return the pointer to the next character if its part of the C-string.

Rfc822Date

This function converts a time_t into its RFC822 representation.

/* Render seconds since 1970 as an RFC822 date string.  Return
** a pointer to that string in a static buffer.
*/
static char *Rfc822Date(time_t t){
  struct tm *tm;
  static char zDate[100];
  tm = gmtime(&t);
  strftime(zDate, sizeof(zDate), "%a, %d %b %Y %H:%M:%S %Z", tm);
  return zDate;
}

The interesting (in a bad way) is that it returns a pointer to a statically allocated memory. So any time you make 2 calls to this bad boy, just remember that the first pointer returned is now invalid :).

One way to solve this is to make the caller of this function provides the bytes to write to. This makes this function free from being responsible for memory; which it shouldn’t.

Testing

Let’s try to use this function and see what we get.

#include <stdio.h>
#include <time.h>

static char *Rfc822Date(time_t t)
{
    struct tm *tm;
    static char zDate[100];
    tm = gmtime(&t);
    strftime(zDate, sizeof(zDate), "%a, %d %b %Y %H:%M:%S %Z", tm);
    return zDate;
}

int main(void)
{
    time_t t = time(NULL);
    char *zDate = Rfc822Date(t);
    const int r = printf("%s\r\n", zDate);
    return r < 0 ? -1 : 0;
}

Compiling and running this small program yields,

Sat, 12 Jun 2021 16:24:15 GMT

References

1. strftime
2. gmtime

DateTag

Nothing really special about this function. It just appends the zTag and the Rfc822Date of the given time. The result is then printed to stdout.

/*
** Print a date tag in the header.  The name of the tag is zTag.
** The date is determined from the unix timestamp given.
*/
static int DateTag(const char *zTag, time_t t){
  return printf("%s: %s\r\n", zTag, Rfc822Date(t));
}

References

1. printf

ParseRfc822Date

This function is quite complicated. I don’t pretend to know half of how it works. However, as the comment said, its supposed to parse an RFC822 formatted strings. It just appends the zTag and the Rfc822Date of the given time.

/*
** Parse an RFC822-formatted timestamp as we'd expect from HTTP and return
** a Unix epoch time. <= zero is returned on failure.
*/
time_t ParseRfc822Date(const char *zDate){
  int mday, mon, year, yday, hour, min, sec;
  char zIgnore[4];
  char zMonth[4];
  static const char *const azMonths[] =
    {"Jan", "Feb", "Mar", "Apr", "May", "Jun",
     "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"};
  if( 7==sscanf(zDate, "%3[A-Za-z], %d %3[A-Za-z] %d %d:%d:%d", zIgnore,
                       &mday, zMonth, &year, &hour, &min, &sec)){
    if( year > 1900 ) year -= 1900;
    for(mon=0; mon<12; mon++){
      if( !strncmp( azMonths[mon], zMonth, 3 )){
        int nDay;
        int isLeapYr;
        static int priorDays[] =
         {  0, 31, 59, 90,120,151,181,212,243,273,304,334 };
        isLeapYr = year%4==0 && (year%100!=0 || (year+300)%400==0);
        yday = priorDays[mon] + mday - 1;
        if( isLeapYr && mon>1 ) yday++;
        nDay = (year-70)*365 + (year-69)/4 - year/100 + (year+300)/400 + yday;
        return ((time_t)(nDay*24 + hour)*60 + min)*60 + sec;
      }
    }
  }
  return 0;
}

Given how complex the notion of timezones and time itself, I bet this function is broken. Its much easier to just rely on the standard libc function — and likely more reliable. See the C code here to see parsing dates back and forth,

References

1. gmtime
2. strftime
3. strptime
4. time
5. memset
6. memcmp

TestParseRfc822Date

This function only exists for testing purposes. I wonder how plausible it is to test all the possible values by forking a bunch of process…

/*
** Test procedure for ParseRfc822Date
*/
void TestParseRfc822Date(void){
  time_t t1, t2;
  for(t1=0; t1<0x7fffffff; t1 += 127){
    t2 = ParseRfc822Date(Rfc822Date(t1));
    assert( t1==t2 );
  }
}

To show that its only used once,

hbina.github.io on  master [?⇡]
❯ cat ./static/althttpd/althttpd.c | rg --line-number 'TestParseRfc822Date'
666:void TestParseRfc822Date(void){
2425:      TestParseRfc822Date();

References

1. assert

StartResponse

The way this program generates the response to HTTP requests is by printing to stdout. This function will setup the HTTP request header with the protocol and the status. It will also do some additional stuff based on the status response.

/*
** Print the first line of a response followed by the server type.
*/
static void StartResponse(const char *zResultCode){
  time_t now;
  time(&now);
  if( statusSent ) return;
  nOut += printf("%s %s\r\n", zProtocol, zResultCode);
  strncpy(zReplyStatus, zResultCode, 3);
  zReplyStatus[3] = 0;
  if( zReplyStatus[0]>='4' ){
    closeConnection = 1;
  }
  if( closeConnection ){
    nOut += printf("Connection: close\r\n");
  }else{
    nOut += printf("Connection: keep-alive\r\n");
  }
  nOut += DateTag("Date", now);
  statusSent = 1;
}

One thing to notice that is that if the first digit in the reply status is >=4, it will append the HTTP header with Connection: close. It probably means that if there’s any issue with the request, the server will just close the connection.

Let’s see how its being used,

hbina.github.io on  master [!?⇡]
❯ cat ./static/althttpd/althttpd.c | rg --line-number 'StartResponse'
677:static void StartResponse(const char *zResultCode){
700:  StartResponse("404 Not Found");
716:  StartResponse("403 Forbidden");
732:  StartResponse("401 Authorization Required");
748:  StartResponse("500 Error");
783:  StartResponse("500 CGI Configuration Error");
799:  StartResponse("500 Server Malfunction");
821:      StartResponse("301 Permanent Redirect");
824:      StartResponse("308 Permanent Redirect");
827:      StartResponse("302 Temporary Redirect");
1317:    StartResponse("304 Not Modified");
1329:    StartResponse("206 Partial Content");
1337:    StartResponse("200 OK");
1389:      StartResponse("302 Redirect");
1424:    StartResponse("206 Partial Content");
1432:    StartResponse("200 OK");
1678:    StartResponse("400 Bad Request");
1703:    StartResponse("501 Not Implemented");
1902:      StartResponse("500 Request too large");
1920:      StartResponse("500 Cannot create /tmp file");

As you can see, these are all HTTP headers. All of them are also statically allocated, I think its worth creating a list of possible HTTP headers and use that. Some extra safety!

Another interesting to note is that if the browser uses HTTP/2, this response header can/should be considered malformed according to the spec.

References

1. HTTP Response Header

NotFound

This is just a generic “404 NOT FOUND” template. The only 2 values that it need to complete the document is:

1. lineno which depends on the caller of this function. As I said in previous entries, I think every compiler have a C macro for this already.
2. zScript which is the document that the user is attempting to retrieve. This is probably the string after the domain in the URL.

/*
** Tell the client that there is no such document
*/
static void NotFound(int lineno){
  StartResponse("404 Not Found");
  nOut += printf(
    "Content-type: text/html; charset=utf-8\r\n"
    "\r\n"
    "<head><title lineno=\"%d\">Not Found</title></head>\n"
    "<body><h1>Document Not Found</h1>\n"
    "The document %s is not available on this server\n"
    "</body>\n", lineno, zScript);
  MakeLogEntry(0, lineno);
  exit(0);
}

Note that before ending the process, it will create a log entry. It’s used in quite a few places,

hbina.github.io on  master [!?]
❯ cat static/althttpd/althttpd.c | rg --line-number 'NotFound'
699:static void NotFound(int lineno){
917:    NotFound(150);  /* LOG: Cannot open -auth file */
943:        NotFound(160);  /* LOG:  http request on https-only page */
958:      NotFound(180);  /* LOG:  malformed entry in -auth file */
1327:  if( in==0 ) NotFound(480); /* LOG: fopen() failed for static content */
1687:  if( zScript[0]!='/' ) NotFound(210); /* LOG: Empty request URI */
1860:        NotFound(260);  /* LOG: Disallowed referrer */
1962:      NotFound(300); /* LOG: Path element begins with "." or "-" */
1974:    NotFound(310); /* LOG: URI does not start with "/" */
1977:    NotFound(320); /* LOG: URI too long */
1980:    NotFound(330);  /* LOG: Missing HOST: parameter */
1982:    NotFound(340);  /* LOG: HOST parameter too long */
2007:        NotFound(350);  /* LOG: *.website permissions */
2049:      if( stillSearching ) NotFound(380); /* LOG: URI not found */
2054:        NotFound(390);  /* LOG: File not readable */
2071:        NotFound(400); /* LOG: URI is a directory w/o index.html */
2211:    NotFound(460); /* LOG: Excess URI content past static file name */

References

1. printf
2. exit

Forbidden

Another HTTP response template. This time its to indicate that the request is forbidden.

/*
** Tell the client that they are not welcomed here.
*/
static void Forbidden(int lineno){
  StartResponse("403 Forbidden");
  nOut += printf(
    "Content-type: text/plain; charset=utf-8\r\n"
    "\r\n"
    "Access denied\n"
  );
  closeConnection = 1;
  MakeLogEntry(0, lineno);
  exit(0);
}

What is Forbidden?

I think its amusing to look at the usage of this function. It’s being used at several places,

hbina.github.io on  master took 4s
❯ cat ./static/althttpd/althttpd.c | rg --line-number 'Forbidden'
715:static void Forbidden(int lineno){
716:  StartResponse("403 Forbidden");
1764:        Forbidden(230); /* LOG: Referrer is devids.net */
1778:        Forbidden(240);  /* LOG: Illegal content in HOST: parameter */
1838:        Forbidden(250);  /* LOG: Disallowed user agent */
1846:      Forbidden(251);  /* LOG: Disallowed user agent (20190424) */

Forbidden Referer

HTTP headers have a “referer” value when making requests so that the server can know who made the request. For whatever reason, the author decides to forbid anyone from devids.net!

    }else if( strcasecmp(zFieldName,"Referer:")==0 ){
      zReferer = StrDup(zVal);
      if( strstr(zVal, "devids.net/")!=0 ){ zReferer = "devids.net.smut";
        Forbidden(230); /* LOG: Referrer is devids.net */
      }

Forbidden Host

When parsing the Host header value, the server will also reply with forbidden if there’s illegal content in it. We will cover what sanitizeString is later.

    }else if( strcasecmp(zFieldName,"Host:")==0 ){
      int inSquare = 0;
      char c;
      if( sanitizeString(zVal) ){
        Forbidden(240);  /* LOG: Illegal content in HOST: parameter */
      }

Forbidden Agents

There’s also a list of disallowed agents. I am not sure why half of these are in here… I guess its just what the author had accumulated after over 20 years of running this program.

if( zAgent ){
    const char *azDisallow[] = {
      "Windows 9",
      "Download Master",
      "Ezooms/",
      "HTTrace",
      "AhrefsBot",
      "MicroMessenger",
      "OPPO A33 Build",
      "SemrushBot",
      "MegaIndex.ru",
      "MJ12bot",
      "Chrome/0.A.B.C",
      "Neevabot/",
      "BLEXBot/",
    };
    size_t ii;
    for(ii=0; ii<sizeof(azDisallow)/sizeof(azDisallow[0]); ii++){
      if( strstr(zAgent,azDisallow[ii])!=0 ){
        Forbidden(250);  /* LOG: Disallowed user agent */
      }
    }

Forbidden Spiders

This one is disabled and its actually quite new! I am not sure what this is? I suppose there’s a specific misbehaving/malicious website crawler attack the author’s server at that time.

#if 0
    /* Spider attack from 2019-04-24 */
    if( strcmp(zAgent,
            "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 "
            "(KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36")==0 ){
      Forbidden(251);  /* LOG: Disallowed user agent (20190424) */
    }
#endif

References

1. printf.
2. exit.
3. HTTP Header Referer.
4. HTTP Header Host.
5. HTTP Header User-Agent.

NotAuthorized

Yet another HTTP response template. This time its to indicate that the request is unauthorised to access that resource.

/*
** Tell the client that authorization is required to access the
** document.
*/
static void NotAuthorized(const char *zRealm){
  StartResponse("401 Authorization Required");
  nOut += printf(
    "WWW-Authenticate: Basic realm=\"%s\"\r\n"
    "Content-type: text/html; charset=utf-8\r\n"
    "\r\n"
    "<head><title>Not Authorized</title></head>\n"
    "<body><h1>401 Not Authorized</h1>\n"
    "A login and password are required for this document\n"
    "</body>\n", zRealm);
  MakeLogEntry(0, 110);  /* LOG: Not authorized */
}

There’s nothing much to be said about this function. So let’s take a look at its usage. There’s only 1 of them.

hbina.github.io on  master [?]
❯ cat ./static/althttpd/althttpd.c | rg --line-number 'NotAuthorized'
731:static void NotAuthorized(const char *zRealm){
964:  NotAuthorized(zRealm);

Configuring Authorization

The function that made the call is CheckBasicAuthorization. This function parses an authorization file that may or may not exist in the directory that the resources is in. One of the parameters that can be added in this configuration file is realm. The documentation says,

**    *  "realm TEXT" sets the realm to TEXT.

So while parsing this configuration file, we assign the “TEXT” above to zRealm here,

    if( strcmp(zFieldName, "realm")==0 ){
      zRealm = StrDup(zVal);

But zRealm is never used!

hbina.github.io on  master [?]
❯ cat ./static/althttpd/althttpd.c | rg --line-number 'zRealm'
731:static void NotAuthorized(const char *zRealm){
740:    "</body>\n", zRealm);
910:  char *zRealm = "unknown realm";
930:      zRealm = StrDup(zVal);
964:  NotAuthorized(zRealm);

So I assume its just a way for the author to determine what kind of files were being requested.

According to the actual HTTP/1.0 specification for authentication, we can see that realms are basically used to tell the requester that the same realm should require the same kind of authentication. Probably for caching purposes?

So it kinda makes sense that this value is not being used anywhere. It’s more like a guideline, not a strict spec. It depends entirely if the resources are properly configured.

References

1. printf.
2. exit.
3. HTTP Header Referer.
4. HTTP Header Host.
5. HTTP Header User-Agent.

CgiError

Another simple template to indicate CGI error.

/*
** Tell the client that there is an error in the script.
*/
static void CgiError(void){
  StartResponse("500 Error");
  nOut += printf(
    "Content-type: text/html; charset=utf-8\r\n"
    "\r\n"
    "<head><title>CGI Program Error</title></head>\n"
    "<body><h1>CGI Program Error</h1>\n"
    "The CGI program %s generated an error\n"
    "</body>\n", zScript);
  MakeLogEntry(0, 120);  /* LOG: CGI Error */
  exit(0);
}

Its only being used once,

hbina.github.io on  master [!?]
❯ cat static/althttpd/althttpd.c | rg --line-number 'CgiError'
747:static void CgiError(void){
2197:      CgiError();

Timeout

A function that will be called to handle timeouts. AFAIK, the server is set up such that each request must be handled in a set amount of time. Beyond that, the server will just timeout.

/*
** This is called if we timeout or catch some other kind of signal.
** Log an error code which is 900+iSig and then quit.
*/
static void Timeout(int iSig){
  if( !debugFlag ){
    if( zScript && zScript[0] ){
      char zBuf[10];
      zBuf[0] = '9';
      zBuf[1] = '0' + (iSig/10)%10;
      zBuf[2] = '0' + iSig%10;
      zBuf[3] = 0;
      strcpy(zReplyStatus, zBuf);
      MakeLogEntry(0, 130);  /* LOG: Timeout */
    }
    exit(0);
  }
}

There’s 2 things that I am unsure here.

1. Why do we need 10 bytes when we only use 4 of them?
2. Why is the the error code here 9XX when there’s already a HTTP request code for this. It appears to be for logging purposes only, which is weird because its only enabled when debugFlag is false.

Let’s see how it’s bein used,

hbina.github.io on  master [!?]
❯ cat static/althttpd/althttpd.c | rg --line-number 'Timeout'
333:static int useTimeout = 1;       /* True to use times */
764:static void Timeout(int iSig){
773:      MakeLogEntry(0, 130);  /* LOG: Timeout */
1352:  if( useTimeout ) alarm(30 + pStat->st_size/1000);
1381:  if( useTimeout ){
1657:  signal(SIGALRM, Timeout);
1658:  signal(SIGSEGV, Timeout);
1659:  signal(SIGPIPE, Timeout);
1660:  signal(SIGXCPU, Timeout);
1661:  if( useTimeout ) alarm(15);
1930:    if( useTimeout ) alarm(15 + len/2000);
1939:  if( useTimeout ) alarm(10);
2224:  if( useTimeout ) alarm(30);
2417:        useTimeout = 0;
2540:INSERT INTO xref VALUES(130,'Timeout');

So it seems to register itself to a bunch of signals and that’s pretty much it.

References

1. signal.
2. HTTP Status Request Timeout.

CgiScriptWritable

Just another HTTP response template for CGI errors.

/*
** Tell the client that there is an error in the script.
*/
static void CgiScriptWritable(void){
  StartResponse("500 CGI Configuration Error");
  nOut += printf(
    "Content-type: text/plain; charset=utf-8\r\n"
    "\r\n"
    "The CGI program %s is writable by users other than its owner.\n",
    zRealScript);
  MakeLogEntry(0, 140);  /* LOG: CGI script is writable */
  exit(0);
}