Add tests for MSC2716 and backfilling history

[MadLittleMods]

This PR is a follow-up to #74 which adds support for application services in Complement blueprints.

Add tests for MSC2716 and backfilling history

• Synapse PR: Add endpoints for backfilling history (MSC2716) synapse#9247
• TARDIS visualization MR: Draft: Synapse and MSC2716 edits (explore the DAG) tardis#1

Dev notes

• https://github.com/matrix-org/complement/blob/master/ONBOARDING.md
• https://pkg.go.dev/github.com/docker/docker/client
• Create new file in a container: https://stackoverflow.com/q/43376740/796832

contextRes := alice.MustDoFunc(t, "GET", []string{"_matrix", "client", "r0", "rooms", roomID, "context", eventStar}, client.WithContentType("application/json"), client.WithQueries(url.Values{
	"limit": []string{"100"},
}))
contextResBody := client.ParseJSON(t, contextRes)
logrus.WithFields(logrus.Fields{
	"contextResBody": string(contextResBody),
}).Error("context res")

Previous code before functional client.go

contextRes := alice.MustDoRaw(t, "GET", []string{"_matrix", "client", "r0", "rooms", roomID, "context", eventStar}, nil, "application/json", url.Values{
	"limit": []string{"100"},
})
contextResBody := client.ParseJSON(t, contextRes)
logrus.WithFields(logrus.Fields{
	"contextResBody": string(contextResBody),
}).Error("context res")

---

Always show server logs even if the test passes: Change this line in internal/docker/deployment.go#L34 so it always passes true:

--- a/internal/docker/deployment.go
+++ b/internal/docker/deployment.go
@@ -31,7 +31,7 @@ type HomeserverDeployment struct {
 // will print container logs before killing the container.
 func (d *Deployment) Destroy(t *testing.T) {
        t.Helper()
-       d.Deployer.Destroy(d, t.Failed())
+       d.Deployer.Destroy(d, true) // t.Failed()
 }

---

Create random string:

• https://stackoverflow.com/a/59457748/796832
• https://stackoverflow.com/a/22892181/796832

hsTokenBytes := make([]byte, 32)
_, err := rand.Read(hsTokenBytes)
if err != nil {
	return bp, err
}

var hsToken string = hex.EncodeToString(hsTokenBytes)

---

Make it possible to read the response body multiple times

body := client.ParseJSON(t, res)
// Since the original body can only be read once, create a new one from the body bytes we just read
res.Body = ioutil.NopCloser(bytes.NewBuffer(body))

// ready to read the body again from res

Hook up Element (Matrix client) to the Complement server after a test runs

Because PublishAllPorts: true, is set when the container is created, the homeserver is available to point a Matrix client against.

1. In your test comment out defer deployment.Destroy(t) and replace with defer time.Sleep(2 * time.Hour) to keep the homeserver running after the tests complete
2. Start the Complement tests (./scripts-dev/complement.sh)
3. Save the Element config as /Users/eric/Downloads/riot-complement-config.json and replace the port with the port you see from docker ps (docker ps -f name=complement_ to just filter to the Complement containers)

   {
     "default_server_config": {
       "m.homeserver": {
         "base_url": "http://localhost:55449",
         "server_name": "my.complement.host"
       }
     },
     "brand": "Element"
   }

4. Start up Element (your matrix client)

   docker run -it --rm \
       --publish 7080:80 \
       --volume /Users/eric/Downloads/riot-complement-config.json:/app/config.json \
       --name riot-complement \
       vectorim/riot-web:v1.7.8
   

5. Now you can visit http://localhost:7080/ and register a new user and explore the rooms from the test output

Access database in Complement test Docker container:

$ docker exec -it complement_1_hs_with_application_service.hs1_2 /bin/bash

$ apt-get update && apt-get install -y sqlite3

$ sqlite3
> .open /conf/homeserver.db

TODO

• Define the client as an appservice
• Add tests for healing state, https://github.com/matrix-org/matrix-doc/pull/2716/files#r559846587
  • Maybe not heal state, Add endpoints for backfilling history (MSC2716) synapse#9247 (comment)
• Add test to ensure only appservice can use the ?prev_event query param
• Add test for "If an unrecognised event ID is specified as a prev_event, the request fails with a 404."
• Add test for "An event must not have more than 20 prev_events."
  • This is enforced by Synapse but would be fine to add a test. Can add in another PR
• Can we support historical state events as the spec wants? PUT /_matrix/client/r0/rooms/{roomId}/state/{eventType}/{stateKey}
  • We've moved to a batch send endpoint
• Add test for inserting historical messages in a room from a user that isn't already in the room
  • I suppose this should be a federation test and testing /backfill essentially
  • Need to make sure the join and membership events are there
  • Make sure avatar and displayName work
  • As an application service, can use the ?user_id query parameter to act as a virtual user
• Add federation test where hs2 is already joined to the room, historical messages are added, scrollback shows the messages
• Add federation test where hs2 is already joined to the room and has backfilled some historical messages. Then more historical messages are added and scrollback is able to show the new historical messages
• Would be cool to make a generalized solution to make a TARDIS visualization for any test run in Complement (after hook to save a file which we can view in TARDIS)
• Create PR to define namespace regexes, Add tests for MSC2716 and backfilling history #68 (comment) -> Allow application service YAML to be customised #148

[MadLittleMods]

@kegsay Can you give this another look?

I don't want to keep adding to it when we were so close to merge. If you only want to approve up to a certain point, I can create another branch with all of the updates and we can merge at any commit. (trying to avoid the conflicts with juggling many branches)

[kegsay]

Yep sorry, I'll look at this again

[kegsay]

Let's merge this now.

[MadLittleMods]

Thanks for the review @kegsay 🦓 Lots of great improvements!

The functional client you refactored feels a lot better too!