How to Run a Local Alternative to a Stable Service Container?

[utkonos]

This is in the context of the Docker appliance deployment type. After making some small changes to my fork of the Extract service:
https://github.com/utkonos/assemblyline-service-extract

I want to test this one by replacing the Extract service container in the deployment. I have been trying the following:

git clone https://github.com/utkonos/assemblyline-service-extract.git
cd assemblyline-service-extract
docker build -t cccs/assemblyline-service-extract:custom .
docker run -dp 32000:5000 --restart=always --name registry registry
docker tag cccs/assemblyline-service-extract:custom localhost:32000/cccs/assemblyline-service-extract:custom
docker push --all-tags localhost:32000/cccs/assemblyline-service-extract

Then I modify the line in deployments/assemblyline/bootstrap-compose.yaml: services.service_extract.image to localhost:32000/cccs/assemblyline-service-extract:custom

But the container is not starting and not listed by docker ps

What am I missing? I didn't notice content in the documentation about this process, but if I missed something please let me know.

[utkonos]

@cccs-rs is there documentation for the proper process for adding a feature to a default service (Extract in this case)?

The dependency in the extract service that handles NSIS installers is currently not fully working. Installers that use bzip2 compression are not being fully extracted. The script (.nsi) is missing from these because the library being used for NSIS extraction uses a SWIG wrapper around the source code for NSIS's bzip2 implementation. But that SWIG wrapper is broken. To solve this problem, I wrote a pure Python implementation of the NSIS fork of bzip2:
https://github.com/utkonos/nsisunbz2

I incorporated it into an RE framework I'm working on for NSIS installers:
https://github.com/utkonos/rensis

I used these two to fix up the NSIS extraction process in the Extract service. But I am not able to see the Extract service running as above. Any idea what the problem is?

[cccs-rs]

I'm going to tag @gdesmar, the maintainer of the Extract service, as they'll be able to help with analysis of NSIS installers.

In response to your initial question, if you want to test your changes against a sample you can use RunServiceOnce which is documented here and if you're performing development in VSCode, there are tasks for testing your code without building the whole container and deploying to Assemblyline. Because in this case, Extract relies on external packages and tools that make up it's analysis pipeline, I would highly recommend the use of devcontainers like in Zeek which I find easier for service development & troubleshooting bugs produced by samples.

[gdesmar]

I haven't looked at an NSIS file in a long time! 😃
Do you have an example that is not working using the current Extract service and my fork of nrs?
I would be interested to use the code in nsisunbz2 to replace the dependency on SWIG.
Looking at rensis, it looks like the goal is to call run() and then get the content of obj.script_bin? If it is possible to share, I would be interested to know what use you have for that file (or what information can be found in it). From my understanding, if you were to look for a filename of setup.bin, xtinno would be able to give you the same content?
I see that in your fork, you are also extracting binref's version of setup.nsis, I may add it as a backup for my SETUP.nsi, but I feel like my version was more easy to read.
I'm looking forward to learn more about other people's NSIS analysis projects! 😁

[utkonos]

Use any nsis installer that is compressed with bzip2 and it will fail to emit a decompilation from your nrs fork. The reason is something in there is broken either the code itself or the SWIG interface. That piece not working is actaully why I wrote nsisunbz2 in the first place.

What I did for testing is to make reference nsis installers of each possible type: lzma, zlib, bz2 as well as utf8 and not and solid and not and all the combinations. I have tested on hundreds of other files as well.

nrs is a python implementation of 7zip's NSIS decompiler. It arrives at a different format than what binary refinery does. They're both useful in different contexts.

The "setup.bin" is just a naming convention from TotalCommander. that chunk of data is not actually a file. The way NSIS works is it takes the user's text NSIS script and compiles it down to a bytecode representation. That bytecode blob is stored in the "first position" in the NSIS installer's overlay. When the installer is executed, that first position is read into a memory location and then parsed by a set of functions in the installer. it represents all the instructions from the user's installer script. The exact bytecode representation of the installer script is highly valuable because this is something one can write a yara signature against to detect NSIS installer behaviors and features. The decompiled text of the script that both nrs and binary refinery emit, each in their own format, is an approximation of this bytecode. There are definitely errors in both nrs and binref, so the best place to target a detection is this bytecode blob because it represents ground truth.

Long story short: the binary original, your nrs text decompile, and binref's text decompile are all three valuable for different reasons when reversing an NSIS installer.

I'm giving a talk on reversing NSIS installers at the end of the summer too. I'll make sure to let you all know.

[utkonos]

Another small wrinkle: the latest yara-python from two days ago has some bad packages on pypi. They're missing dependencies. So, rensis is not working with that version. I am making some changes to lock the yara-python verison to 4.5.0 until someone pushes packages to pypi that are not broken.
VirusTotal/yara-python#277

[utkonos]

Scratch that last one. The problem with the wheels is only on macOS. Linux works just fine.

[gdesmar]

Then I modify the line in deployments/assemblyline/bootstrap-compose.yaml: services.service_extract.image to localhost:32000/cccs/assemblyline-service-extract:custom

@cccs-rs Every time I see localhost in a container path such as this one, I'm afraid that's the issue. 😅
Is this another one of those "Scalers runs in the container so it won't find it at localhost" issue?

[cccs-rs]

Possibly, although it's mentioned that the container doesn't start up when running the bootstrap which is odd because Scaler shouldn't be at play then. But I think even then, the localhost part of the container image path should be interpreted by Docker so it should know that there's a registry on port 32000 if it's configured.

[utkonos]

Is there a different way to do this? The command docker build -t cccs/assemblyline-service-extract:custom . appears to work correctly. Is there a way that I can use the built container that results from that command with the ones in assemblyline without using a local container registry? (and without uploading to docker's container hub).

[cccs-rs]

There is a launch task that allows you to run the service "live" and it should hook into an existing Assemblyline appliance in Docker.

[utkonos]

Oh? What is the specific command for that? Or can you point me at documentation?

[utkonos]

@cccs-rs @gdesmar Rather than using the localhost registry, I am trying to isolate variables and problems. I have pushed the container with the nsis modifications to docker's hub. I changed ${REGISTRY}cccs/assemblyline-service-extract:${SERVICE_VERSION} to utkonos/assemblyline-service-extract:custom in the file deployments/assemblyline/bootstrap-compose.yaml. However, in the web interface of assemblyline, it doesn't appear correctly.

First, the extract service is showing disabled:

The version is showing this:

Then in the service details, under container, it still shows the wrong container:

If I modify the container image name and try to enable the container, it doesn't work:

It shows my image name now:

But enable does not work. There is now a update button that has appeared:

[utkonos]

Getting closer, but it still doesn't work.

I changed this line in the manifest so that the repository name is correct.

Then I built and pushed using the following:

export VERSION=4.6.0.custom1
docker build -t utkonos/assemblyline-service-extract:$VERSION --build-arg=$VERSION .
docker tag utkonos/assemblyline-service-extract:$VERSION utkonos/assemblyline-service-extract:$VERSION
docker push utkonos/assemblyline-service-extract:$VERSION

The image is there:

Changed the line in bootstrap-compose.yaml:

But then inside AL, the Extract service is still disabled:

The container image is almost correct, but the tag is wrong:

[utkonos]

I tried using exactly the version 4.6.0.stable4

export VERSION=4.6.0.stable4
docker build -t utkonos/assemblyline-service-extract:$VERSION --build-arg=$VERSION .
docker tag utkonos/assemblyline-service-extract:$VERSION utkonos/assemblyline-service-extract:$VERSION
docker push utkonos/assemblyline-service-extract:$VERSION

Bootstrap:

Didn't work:

[utkonos]

@cccs-rs Is there something else I'm doing wrong here?

[cccs-rs]

Looks like the build arg might be off?

- docker build -t utkonos/assemblyline-service-extract:$VERSION --build-arg=$VERSION .
+ docker build -t utkonos/assemblyline-service-extract:$VERSION --build-arg version=$VERSION .

Also worth noting that Assemblyline only recognizes the Development (dev) and Stable (stable) branches for a service. So I don't expect it to recognize custom very well ( ie. 4.6.0.custom1).

[utkonos]

Good catch. It works now:

export VERSION=4.6.0.stable4
docker build -t utkonos/assemblyline-service-extract:$VERSION --build-arg version=$VERSION .
docker tag utkonos/assemblyline-service-extract:$VERSION utkonos/assemblyline-service-extract:$VERSION
docker push utkonos/assemblyline-service-extract:$VERSION

The custom container is running now.

[utkonos]

Just to summarize in one place.

1. Change the docker_config.image field in service_manifest.yml. Here is the location.
2. Build and push using the code snippet below. Change foo to your Docker hub repository name. And change the version to an appropriate number that is compatible with the version of AL that you're deploying to.
3. Change the service_extract.image field in bootstrap-compose.yaml. Make sure to have the version tag here match the version in the previous step. Here is the location.
4. Do everything else like usual.

export VERSION=4.6.0.stable4
docker build -t foo/assemblyline-service-extract:$VERSION --build-arg version=$VERSION .
docker tag foo/assemblyline-service-extract:$VERSION foo/assemblyline-service-extract:$VERSION
docker push foo/assemblyline-service-extract:$VERSION