Understanding the HTTP Protocol

Oh, hey there!

This is the first article of a new series of a series called "Build the System".
On "Build the System", or BTS for short, I want to explore how the technologies we use as web developers
works. I will explain the theory and then I will build the system from the ground up.
I think this will be a great opportunity to also discuss software design, documentation and testing.
To launch this new project I chose to dissect the HTTP protocol.
We use HTTP every day and it's kinda important in web development, wouldn't you say so?

This article is a transcript of a Youtube video I made.

I said that HTTP was a protocol, more specifically a network protocol. A protocol can be defined as:

[...] a set of rules, conventions, and data structures that dictate how devices exchange data
across networks. In other words, network protocols can be equated to languages that two devices must understand
for seamless communication of information, regardless of their infrastructure and design disparities.

It's similar to when you cross a co-worker in the morning and they ask you "How are you?" and you answer

• "Fine, and you?" even though you're not "fine" at all...

We know that HTTP is a protocol because that's what the P stands for. The other letters stands for HyperText Transfer...
Now we know that HTTP is a protocol used to Transfer HyperText... 🤷

The word HyperText kinda means that it's text beyond text... Or text that overcame the limitation or constraints of
text...
The word was first used to refer to text that could contain hyperlinks; to link various documents together.
Today, HTTP can be used for a lot more than text -- so the guy who coined the words hypertext and hyperlink, also came
up with "hypermedia"... But, I guess HMTP didn't sound right?

So yeah...

The Hypertext Transfer Protocol (HTTP) is an application layer protocol for distributed, collaborative, hypermedia
information systems.

Alright, let's just jump into it. Here I have two terminal sessions open. One will be the server, and the second
will be the client. The client will make a request and the server will "respond".

The first terminal will be used to listen to a TCP connection and write a response. The second terminal will be used
to make a HTTP request.

By the way, TCP is:

[...] one of the main protocols of the Internet protocol suite. ... a connection between client and
server is established before data can be sent.

I might cover TCP/IP further in another article, but all you need to know is that it provides reliable, ordered, and
error-checked delivery of a stream of bytes.

To handle the server-side of things, I will use netcat which is a tool to make or listen to arbitrary TCP and UDP
connections.
On the server's teminal I will type nc -k -l 8080 to listen to incoming connections to the port 8080 of my computer.
On the client's terminal I will initiate a HTTP request using the curl command.

$ curl localhost:8080

On the server side, you'll see our first request.

GET / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.64.1
Accept: */*

Let's analyze it a little.

The first line is usually refered to as the "request-line".
The first part is called the method. It defines what action to perform on the resource. GET is often the default.
The second part is the request-uri or path. It identifies the resource that is being requested. / being the root. "What
the root is" is open to interpretation.
Finally, the last part defines the version of the protocol that is used. For now we'll focus on version 1.1 Which is
slowly being phased out in favour of version 2 or 3. But they build on top of one another.
I'll cover HTTP/2 or 3 if there is interest.

Don't hesitate to like, comment and subscribe to let me know you want to learn more about HTTP!

Anyway... Every line of the message are delimited by a CRLF sequence or end-of-line (EOL) marker.
CRLF stands for "Carriage Return and Line Feed". They are usually represented by ASCII 13 and 10 or \r\n.

After the "request-line", the request may or may not come with headers. Headers can be seen as metadata on the request.
They are represented by a field name and value separated by colon and, usually, a space.

In this example, the headers tells us that:

- the request was made to `localhost` which can be useful to know if the server acts as a gateway or proxy;
- the request was sent by the `curl` program;
- the client would accept anything as a response;

Finally, an empty line is sent either ending the request or beginning the body stream.

In this example, there is no body -- the client is waiting for a response.

I will type-in the response...

HTTP/1.1 204 No Content

The response starts with the status-line.
The first part confirms the protocol and the version.
The second part is the status code. I'm sure you know a few already, like 200 or the infamous 404.
Then finally, the reason phrase.
Like for the request, the status-line may be followed by headers and then it must send an empty line and potentially a
body stream.

With this response, we're telling the client that the request was successfully received, understood and accepted but
there will be no content.
This completes the request and the client terminates the connection.

Let's do two more examples...

On the client's terminal I will send a new request:

$ curl localhost:8080/ping

On the server side we can read the GET request to /ping. I will simply respond with a short message but this time,
I will include some headers.

HTTP/1.1 200 OK
Content-Length: 4
Content-Type: text/plain

pong

The Content-Length header will warn the client to expect 4 bytes of data; the Content-Type headers will inform that
the message is just some plain old text. Then following an empty line I can type my message "pong".
This completes the request and the client terminates the connection.

Okay, let's do one last example.

This time, the client will send a body message.

curl localhost:8080/users -H "Content-Length: 23" -H "Content-Type: application/json" -d "{\"fullName\":\"John Doe\"}"

POST /users HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.64.1
Accept: */*
Content-Type: application/json
Content-Length: 23

{"fullName":"John Doe"}

Server side, we can read the request-line, we have a POST request to the /users resource. The Content-Type is
application/json, and the server should expect 23 bytes. After an empty line, we can read the JSON.

HTTP/1.1 201 Created
Location: localhost:8080/users/1
Content-Length: 0

The server respond with a status code of 201 which means that it created the resource.
Then the server tells the client where it can find the new resource with the Location header. Finally, the response
states that there will be no incoming message.
This completes the request and the client terminates the connection.

I think this is a good place to end this introduction to the HTTP protocol.
I hope I was successful on shedding some light on the mysteries of HTTP. Or at least make you curious to learn more...
On the next article, we will start implementing a HTTP server.
We will learn how to decode a streams of bytes into an actionable request and then respond appropriately.
I will also try to offer a reliable way to test our code.
On the following article we will prepare our server to receive and send static files to and from the file system.
Then, on the next article we will implement data streams which is necessary to deal with larger files or to upgrade to
HTTP/2.
Finally, if all goes to plan, the fifth article will focus on building a programmatic router, think Express, Koa or Oak.

At any rate, if this article was useful to you, hit the like button, leave a comment to let me know or best of all,
follow if you haven't already!

Ok bye now...