Upstream to fq?

[wader]

Hey! author of fq here. Accidentally found this repo and wondered if the code is or could be mede generic enough to be upstreamed as a RDP decoder? possibly also add the plumbing to make it decode RDP in pcap files? Happy to help of even give it a try myself

[wader]

Oh also found the black hat presentation now, congrats! nice to see fq comes to use

[obilodeau]

Hi! Glad you found this 😁

Our own storage format for pyrdp replay files and the RDP network format are a bit different. We avoid storing unnecessary encapsulation and we skip some message types (clipboard file i/o for example). That said, it would probably be possible to support both if fq's architecture allows code re-use. I'm sure it does but I don't have much experience with Go so it would not be obvious to me how it needs to be done.

The biggest problem I see is that regular RDP as captured in Pcaps is TLS encrypted. PyRDP does store TLS master secrets to an external log file so something could be done with the fq tls module (just found out about it). Something like pcap > tls > rdp.

Obtaining decryptable Pcaps outside of PyRDP is possible but difficult. Ciphers needs to be weaken to avoid EDH and private key must be extracted from the OS. Steps are documented here: https://github.com/GoSecure/pyrdp/blob/main/docs/debugging-recipes.adoc#decrypt-a-regular-non-intercepted-by-pyrdp-rdp-session.

Feel free to integrate this in upstream fq if you have time. I can help with advice and sample files (both pcaps and pyrdp). Once the integration is done I will happily contribute upstream as I add supported fields for my research needs.

[wader]

Our own storage format for pyrdp replay files and the RDP network format are a bit different. We avoid storing unnecessary encapsulation and we skip some message types (clipboard file i/o for example). That said, it would probably be possible to support both if fq's architecture allows code re-use. I'm sure it does but I don't have much experience with Go so it would not be obvious to me how it needs to be done.

I see! it should probably be possible to reuse some things i think

The biggest problem I see is that regular RDP as captured in Pcaps is TLS encrypted. PyRDP does store TLS master secrets to an external log file so something could be done with the fq tls module (just found out about it). Something like pcap > tls > rdp.

Yeap it can decrypt tls 1.0, 1.1 and 1.2 if keylog is provided as an cli option and it shouldn't be that hard to also make a decoder (like pcapng ) able to pass keylog if available to the tls decoder.

Obtaining decryptable Pcaps outside of PyRDP is possible but difficult. Ciphers needs to be weaken to avoid EDH and private key must be extracted from the OS. Steps are documented here: https://github.com/GoSecure/pyrdp/blob/main/docs/debugging-recipes.adoc#decrypt-a-regular-non-intercepted-by-pyrdp-rdp-session.

Feel free to integrate this in upstream fq if you have time. I can help with advice and sample files (both pcaps and pyrdp). Once the integration is done I will happily contribute upstream as I add supported fields for my research needs.

Great! hope i can get some time soon to give a quick try to see how it looks.

Sample files and keylogs would be much appreciated!

[obilodeau]

I'm at a cybersecurity defense event until Thursday and we have time allocated to upstream this into fq.

For now, I would keep the Pcap portion for later since this is a significant rework of something that exists and that is about to be useful for a data pipeline here. Side note: we would like to avoid the complicated build instructions of having an out-of-fq module.

You mentioned sample files and keylogs. There is testdata here: https://github.com/GoSecure/fq-pyrdp/tree/main/pyrdp/testdata is that enough? It looks like it's not getting automatically used by the testing environment. Any pointers on that?

I'm going to add more feature extraction out of PyRDP captures today. To make the parser more complete.

[obilodeau]

Just found how to run the tests: https://github.com/wader/fq/blob/b0025b64c94aa443e310647a4148c4c8015d7d1c/doc/dev.md#checklist

[wader]

I'm at a cybersecurity defense event until Thursday and we have time allocated to upstream this into fq.

For now, I would keep the Pcap portion for later since this is a significant rework of something that exists and that is about to be useful for a data pipeline here. Side note: we would like to avoid the complicated build instructions of having an out-of-fq module.

Ok 👍 so will be format decoder that will be run directly on already extracted tcp stream etc?

You mentioned sample files and keylogs. There is testdata here: https://github.com/GoSecure/fq-pyrdp/tree/main/pyrdp/testdata is that enough? It looks like it's not getting automatically used by the testing environment. Any pointers on that?

I'm going to add more feature extraction out of PyRDP captures today. To make the parser more complete.

Just found how to run the tests: https://github.com/wader/fq/blob/b0025b64c94aa443e310647a4148c4c8015d7d1c/doc/dev.md#checklist

For most decoder the tests work by putting a few samples files into the testdata directory, create/generate <sample>.fqtest file for each sample with some suitable fq command like $ fq dv <sample> (maybe with -d <formatname> if format is not probe-able), run go testwith -update to rewrite the .fqtest files with expected output, have a look if things look sane, done 🥳

Let me know how it goes!

[obilodeau]

Ok 👍 so will be format decoder that will be run directly on already extracted tcp stream etc?

Not exactly. It's a dump of the RDP wire format as saved by PyRDP. Doesn't contain all of RDP (this is why a common parser will be challenging even after pcap decryption) but it contains enough to replay screen content, keystrokes and mouse movement. These files are meant to be replayed with the pyrdp-player (see screenshot below).

The reason we wrote a parser is to reprocess these files and look at protocol-specific portions of them at scale. For example, extract all keyboard layout information from all capture files we have.

Our current test file would add some weight to your repo:

> ls -lh format/pyrdp/testdata/*
-rw-rw-r-- 1 obilodeau obilodeau 7.0M May 29 16:45 format/pyrdp/testdata/test.fqtest
-rw-rw-r-- 1 obilodeau obilodeau  24M May 31 14:39 format/pyrdp/testdata/test.pyrdp

Is this too much or should I proceed with a PR?

[wader]

Not exactly. It's a dump of the RDP wire format as saved by PyRDP. Doesn't contain all of RDP (this is why a common parser will be challenging even after pcap decryption) but it contains enough to replay screen content, keystrokes and mouse movement. These files are meant to be replayed with the pyrdp-player (see screenshot below).

I see, so to use this a user would have dump rdp traffic and use pyrdp to decrypt first? trying to figure how useful this will be to some other fq user. Don't want to waste your time if it turns out to be a bad fit :)

If possible could you do some kind of draft PR so i get a better picture?

Our current test file would add some weight to your repo:

> ls -lh format/pyrdp/testdata/*
-rw-rw-r-- 1 obilodeau obilodeau 7.0M May 29 16:45 format/pyrdp/testdata/test.fqtest
-rw-rw-r-- 1 obilodeau obilodeau  24M May 31 14:39 format/pyrdp/testdata/test.pyrdp

Is this too much or should I proceed with a PR?

Oh hmm that is quite a lot. I don't know much about RDP but it the protocol easy to "slice", ex could one take just part of the stream and still decode it somehow? maybe that could be an option to decrease the size a bit. Another way could be to dump a very short rdp test session but i don't know how much work it is to setup an env to do that?

[obilodeau]

I see, so to use this a user would have dump rdp traffic and use pyrdp to decrypt first? trying to figure how useful this will be to some other fq user. Don't want to waste your time if it turns out to be a bad fit :)

PyRDP does the dumping/decrypting itself, no wireshark is required.

If possible could you do some kind of draft PR so i get a better picture?

Yes, I will.

Another way could be to dump a very short rdp test session but i don't know how much work it is to setup an env to do that?

I'll find something smaller. No problem.

[obilodeau]

Track the upstreaming progress here: wader/fq#959