SDC Decomposition Sprint 2

common-content/en/module/decomposition/chat-application-requirements/index.md

@@ -0,0 +1,50 @@
++++
+title = "Chat application requirements"
+headless = true
+time = 30
+facilitation = false
+emoji= "📝"
+objectives = [
+    "Explain the requirements of our chat application.",
+    "Explain what requirements are out of scope for our chat application.",
+]
++++
+
+You are going to make a chat application which lets multiple users exchange messages.
+
+You've seen a similar application in the quote application. This time, you will be writing all of the code yourself.
+
+Let's think about some requirements for our application:
+* As a user, I can send add a message to the chat.
+* As a user, when I open the chat I see the messages that have been sent by any user.
+* As a user, when someone sends a message, it gets added to what I see.
+* As a user, I can "like" or "dislike" someone's message.
+* When messages are liked or disliked, a count of the likes and dislikes is displayed next to the message.
+* As a user, I can schedule a message to be sent at some time in the future.
+* As a user, I can change the colour messages that I send.
+* As a user, I can make some words in my messages bold, italic, or underlined.
+* As a user, I can indicate that my message is a reply to another message.
+
+We can imagine other requirements too (e.g. reacting with emojis, being able to edit or delete messages, registering exclusive use of a username, ...). We will stick just to the requirements we've listed. In fact, the requirements we've listed are probably more than we have time to implement, so we will need to prioritise them and choose which ones we have time to build. Think about which requirements are _absolutely_ required - we will definitely need to build those!
+
+Because users want to see things, we know we'll need a frontend.
+
+Because multiple users want to be able to share information (messages), we know we'll need a backend for them to communicate via.
+
+### What we already know and what's new
+
+Some of these requirements are similar to the quote server we've already implemented:
+* Adding messages is like adding quotes.
+* Seeing messages is like seeing quotes.
+
+Others are new:
+* Live updates
+* Interacting with a message
+* Scheduling sending messages.
+* Changing colour of messages.
+* Formatting specific parts of a message.
+* Replying to messages.
+
+First let's make a backend and a frontend to do what we already know. This shouldn't take us very long (we know how to do it, and have done it recently). It will give us a useful framework to experiment with the things that are new to us.
+
+If we thought it would take us a long time to do what we already know, we may approach this differently. We would probably try to work out the new things first. Because they may change how we want to do everything. But because it should be quick to do what we do know, we'll start there.

common-content/en/module/decomposition/deploy-chat-application/index.md

@@ -1,10 +1,13 @@
 +++
 title = "Deploying the chat application"
 headless = true
-time = 30
-emoji= "📖"
-[objectives]
-    1="Deploy all components of a frontend/backend/database application so it can be used on the internet"
+time = 60
+emoji= "➡️"
+objectives = [
+    "Deploy a frontend and backend so it can be used on the internet.",
+]
 +++
 
-### Deploying the chat application
+{{<note type="Exercise">}}
+Deploy your chat application so that both the frontend and backend are on the Internet, can talk to each other, and can be used by people.
+{{</note>}}

common-content/en/module/decomposition/designing-streaming-api/index.md

@@ -0,0 +1,46 @@
++++
+title = "Designing a streaming API"
+headless = true
+time = 30
+facilitation = false
+emoji= "🧑‍🔬"
+objectives = [
+    "Describe the trade-offs of different streaming APIs.",
+]
++++
+
+So far we have a frontend that asks a backend for all of its chat messages, and shows them.
+
+We want the frontend to find out about new messages.
+
+One way we could do this is for our frontend to frequently ask our backend for all of its messages. We could then either re-render the page with all of the messages, or we could work out which messages are new and add them to the UI.
+
+This approach can work, but it is {{<tooltip title="expensive">}}When we use the word expensive, we don't always mean in terms of money. Sometimes we mean something is slow, something will use a lot of memory, or will use a lot of some other resource{{</tooltip>}}. In this case, the reason it's expensive is that it involves sending a lot of data over the network. Often the _same_ data _repeatedly_.
+
+A cheaper approach would be for the client to be able to ask only for the new messages. But this is a more complicated question than it seems. What does "new" mean? New since when?
+
+We can divide approaches to identifying new messages based on whose job it is to answer the question "since when?".
+
+### Remembering "since when" on the client
+
+If the server tells the client, in its response, some identifier for its newest message (e.g. the timestamp, or some ID), the client could ask in its next request "Please give me all messages since this timestamp".
+
+The client would need to remember the last timestamp/ID it has seen.
+
+The server will need to be able to answer that question: "What messages have been sent since this time?".
+
+There are a few ways we can support this on the server:
+1. If we're storing messages in an array, in all our responses we could tell the client what the last index we returned to it was. Then the server could just `slice` the array to get the values to return.
+2. Or if we don't want to expose array indexes (e.g. because perhaps messages can be deleted), we could put a timestamp in every message object, and `filter` our array based on the timestamp we're searching from.
+
+### Remembering "since when" on the server
+
+The server could remember, for each client, what the last message it sent to that client was.
+
+The client would no longer need to remember the last timestamp/ID it had been sent. It would just ask for new messages.
+
+The server would need a way of identifying which client is making a request, and would itself need to remember which messages it had already sent that client.
+
+This simplifies things for the client, but means that the server needs to remember information for each client it has. If there are a lot of clients, this may be a lot of work, and take up a lot of memory. One challenge with this approach is that the server needs to know when it can forget about a client - if it never forgets about clients, it will keep using more and more memory to track this information for every client it's ever had. And we probably can't rely on a client to tell our backend "you can forget about me now".
+
+We will try out both of these approaches in two different frontends to the same backend.

common-content/en/module/decomposition/long-polling/index.md

@@ -0,0 +1,85 @@
++++
+title = "Long-polling"
+headless = true
+time = 90
+facilitation = false
+emoji= "⏳"
+objectives = [
+    "Stream live updates from a server by using long-polling.",
+    "Compare the trade-offs involved with frequent small requests vs long-polling.",
+]
++++
+
+We noticed when we were repeatedly polling for updates that we were making lots of requests, and a lot of their responses were empty.
+
+Another approach we could take is to make an HTTP request, but have the server not respond to it until there is a non-empty response.
+
+One request gets one response, but the server can take some time to make that response.
+
+This is _also_ expensive, but in a different way.
+
+### Different kinds of expensive
+
+Making lots of requests was expensive because establishing connections takes time, CPU, and network resource. And on the server calculating that there were no new messages to respond with took a bit of time.
+
+Keeping requests open for a long time is expensive because it means the server needs to keep all of the resources associated with a request open. The whole time the request is waiting, the server needs to keep the TCP connection open, needs to keep the memory associated with the request, needs to remember that it should respond to it (and how to). It also means that we need to be careful about restarting the server (e.g. if we change its code and want to run the new code) - stopping the server will break any connections that are waiting.
+
+When we were making lots of small quick connections, we could do things like start a new server, shift over all new traffic to it, and turn off the old server. This was easy for us to do because each request was quick, so we knew when we switched to the new server, any requests to the old server would be done quickly, and it wouldn't get any more.
+
+Now that we're using long connections, if we turn on a new server and switch new requests to it, it may still be a while before the old server's requests are finished.
+
+In different contexts, it may be worse to have lots of short requests, or fewer longer requests.
+
+### Changing our code to support long-polling
+
+On the client side, we probably don't need to change anything about our code. Just the requests it makes may take longer to resolve.
+
+On the server side, we will need to change things around. Right now, our `GET` handler _always_ responds based on what it currently knows.
+
+We want to change this.
+
+If we don't have any messages to respond with, we want to wait until we _do_ have something to respond with.
+
+We know `res.send` is a callback function - it's a function we can call when we have a response to send.
+
+Now we don't always want to call `res.send` in our handler. Sometimes we want to say: "When we next get a message, call `res.send` with that message".
+
+This means we need to share our callback between the `GET` handler for asking about new messages, and the `POST` handler for sending new messages.
+
+When the `POST` handler saves a new message, we want to check if there are any `GET` handlers waiting for a message, and call those `GET` handlers' `res.send` callbacks with the new message.
+
+To achieve this, we probably want to make a new piece of state on our server where we can remember which callbacks still need to be called with responses:
+
+```js
+const callbacksForNewMessages = [];
+```
+
+In our `GET` handler, we want to write some code to remember that someone of our callbacks aren't being instantly called, and will need calling in the future:
+
+```js
+if (messagesToSend.length === 0) {
+    // Note: We need to use an arrow function here, rather than just pushing `res.send` directly.
+    // This is because of handling of "this".
+    // You can read about "this" at https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/this
+    callbacksForNewMessages.push((value) => res.send(value));
+} else {
+    res.send(messagesToSend);
+}
+```
+
+and in our `POST` handler, when we are saving a new message we want to run some code to call any callbacks which are still waiting for a response when we get a new message:
+
+```js
+while (callbacksForNewMessages.length > 0) {
+    const callback = callbacksForNewMessages.pop();
+    callback([newMessage]);
+}
+```
+
+Note how we're wrapping `newMessage` in an array - we need to return the same type of information whether we're immediately calling our callback, or doing it later on.
+
+{{<note type="Exercise">}}
+Update your backend to support long-polling.
+
+You may want to make this an optional behaviour (e.g. by adding a command line flag, or allowing a client to opt in to fast- or long-polling based on a query parameter) so that you can easily experiment with both.
+{{</note>}}

common-content/en/module/decomposition/non-live-chat-application/index.md

@@ -1,12 +1,35 @@
 +++
 title = "Making a non-live chat application"
 headless = true
-time = 30
-emoji= "📖"
-[objectives]
-    1="Write and run a frontend, backend, and database"
-    2="Persist data passed from a user into a database"
-    3="Display data from a database to a user"
+time = 60
+emoji= "🚧"
+objectives = [
+    "Write and run a frontend and backend",
+    "Store data passed from a user into a backend",
+    "Display data from a backend to a user",
+]
 +++
 
-### Making a non-live chat application
+{{<note type="Exercise">}}
+Write a frontend and backend for our chat application.
+
+Remember, the two requirements we have to start with are:
+1. As a user, I can send add a message to the chat.
+2. As a user, when I open the chat I see the messages that have been sent by any user.
+
+But soon we'll be adding:
+
+3. As a user, when someone sends a message, it gets added to what I see.
+
+As well as some of the other requirements.
+
+It's useful to remember what we're going to do next - it may impact how we do things now.
+
+But for now, our _main_ focus should be our two first requirements.
+{{</note>}}
+
+{{<note type="Exercise">}}
+Run your frontend and backend together locally.
+
+Test that it works as we expect. Send several messages. Refresh the page. Make sure you see all of the message (in the correct order).
+{{</note>}}

common-content/en/module/decomposition/repeated-client-requests-for-updates/index.md

@@ -0,0 +1,45 @@
++++
+title = "Repeated client requests for updates"
+headless = true
+time = 90
+facilitation = false
+emoji= "🔄"
+objectives = [
+    "Stream live updates from a server by using repeated HTTP polling.",
+]
++++
+
+First let's explore how the client can remember "since when".
+
+{{<note type="Exercise">}}
+Teach your backend how to answer "since when" queries.
+
+You can choose whether you want to use array indexes, timestamps in objects, or some other "since when" tracker.
+
+You can also choose whether you want this to be the same endpoint as your "get all messages" endpoint (with some query parameter), or a separate endpoint.
+{{</note>}}
+
+Now that our backend can answer the question "show me messages since", we can update our frontend to repeatedly ask for new messages, and add them to the UI.
+
+Here is an example function we could write to do this:
+
+```js
+const keepFetchingMessages = async () => {
+    const lastMessageTime = state.messages.length > 0 ? state.messages[state.messages.length - 1].timestamp : null;
+    const queryString = lastMessageTime ? `?since=${lastMessageTime}` : "";
+    const url = `${server}/messages${queryString}`;
+    const rawResponse = await fetch(url);
+    const response = await rawResponse.json();
+    state.messages.push(...response);
+    render();
+    setTimeout(keepFetchingMessages, 100);
+}
+```
+
+It asks for messages since the last seen message (or all messages, if we don't know about any), updates the known messages state, re-renders the page, and then calls itself in 100ms.
+
+This means that within about 100ms of a message being sent, we should know about it and display it.
+
+We only transfer new messages, we're not sending all messages over the network every 100ms. That would be a lot of data.
+
+But we _are_ making a lot of requests. Probably most of those requests get an empty response. And this is in its own way expensive - making a new HTTP connection over and over again takes some compute and network resources. And each time we make a request, our server has to do some work to work out how it should answer.

common-content/en/module/decomposition/single-request-response-limitations/index.md

@@ -0,0 +1,53 @@
++++
+title = "Limitations of a single request/response"
+headless = true
+time = 30
+facilitation = false
+emoji= "✋"
+objectives = [
+    "Identify the limitations of loading a website via a single request and response.",
+    "Give examples of functionality that relies on subsequent data transfer from the server to the client.",
+    "Identify and explain why clients typically initiate requests.",
+]
++++
+
+HTTP is a single-request/single-response protocol.
+
+A client makes a single request, and a server sends a single response.
+
+This leaves no room for updates. In our chat example, we made a request, and the server could tell us all of the messages it knew about when it responded.
+
+If a new message comes in, the server has no way of telling us about it. It already sent its response.
+
+We can imagine a few ways to overcome this limitation, for example:
+1. The client could also the server again.
+2. We could use something _other_ than an HTTP request which allows multiple responses, or bi-directional communication.
+3. The client could ask the server to make an HTTP request to it if there are new updates.
+
+### Limitations of an un-addressable client
+
+The client probably can't ask the server to make an HTTP request to it if there are new updates.
+
+This would effectively ask the client to act as a server, and the server to act as a client.
+
+This is possible! One program can be both a client and a server.
+
+But it is difficult, because to make a connection to a server we need to know an address for it. And most computers running web browsers don't have a public address other computers can use to access them.
+
+If it did have a public address, the client would also need to reserve a port so that when an HTTP request came into the computer, the operating system would know how to give it to the right web-page.
+
+Most web browsers don't have an API for reserving a port and sending a publicly accessible address+port to a server, so we probably can't do this.
+
+### Overcoming these limitations
+
+We've seen that asking the server to make an HTTP request to the client is hard. Let's explore the other two options we mentioned.
+
+#### The client could ask the server again
+
+We've seen that the server can't just send a response to the client without a request. But a frontend can do things after a page first loads. The frontend could ask the backend for any new messages.
+
+#### The client and server could set up bi-directional communication separate from HTTP request/response pairs
+
+Most Web browsers support [the WebSocket API](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API), which allows bi-directional communication.
+
+We will try out both of these solutions.

org-cyf-sdc/content/decomposition/sprints/2/backlog/index.md

@@ -4,6 +4,6 @@ layout = 'backlog'
 emoji= '🥞'
 menu_level = ['sprint']
 weight = 2
-backlog= 'Module-Template'
-backlog_filter='📅 Sprint 1'
+backlog= 'Module-Decomposition'
+backlog_filter='📅 Sprint 2'
 +++