Applications of CTM in medical images

[Himanshunitrr]

Thanks for sharing this I still dont undersand CTM fully but I used image_classification and trained it on NCT-CRC-HE-100K (classes: Adipose (ADI), background (BACK), debris (DEB), lymphocytes (LYM), mucus (MUC), smooth muscle (MUS), normal colon mucosa (NORM), cancer-associated stroma (STR), colorectal adenocarcinoma epithelium (TUM))

Results: (It overfits)

0_attention.gif: (reduced number of frames so that i can attach it here)

accuracies.png:

losses.png:

neural_dynamics_other.png:

neural_dynamics_synch.png

Wanted to know what might be potential applications of CTM in various tasks involving medical images as there are no interpretable methods as of now. More than happy to run experiments under some help/ guidance.

[lukedarlow]

Very interesting that you're doing so! And, thank you for engaging.

Could you share more details of your setup, including the number of internal ticks, the width of the model, and the backbone parameters?

Since this is quite new and experimental research, I can probably help you with sharing my intuitions, and eventually we (the community) can work toward training guidelines that make the most sense.

My immediate first thought is that you are not getting strong emergent dynamics (like those on the gif on the github readme). This sometimes happens with 'simpler' datasets (w.r.t., dataset size, number of classes; e.g., CIFAR-10), with more dynamics emerging over longer training runs or with more challenging setups.

Off the bat, what you can try:

1. A lower learning rate
2. More dropout (set it to 0.3 or so, if you have not) and/or more data augmentation.
3. A more powerful backbone network (try 'resnet18-3'; what are you trying now?)

Thanks again for engaging! I hope that we can find a great configuration for you.

[Himanshunitrr]

Thanks again for the detailed response. Will try your suggestions.

I used the default parameters:

model='ctm',
d_model=512,
dropout=0.0,
backbone_type='resnet18-4',
d_input=128,
heads=4,
iterations=75,
positional_embedding_type='none',
synapse_depth=4,
n_synch_out=512,
n_synch_action=512,
neuron_select_type='random-pairing',
n_random_pairing_self=0,
memory_length=25,
deep_memory=True,
memory_hidden_dims=4,
dropout_nlm=None,
do_normalisation=False,
num_layers=2,
batch_size=64,
batch_size_test=32,
lr=0.001,
training_iterations=100001,
warmup_steps=5000,
use_scheduler=True,
scheduler_type='cosine',
milestones=[8000,
15000,
20000],
gamma=0.1,
weight_decay=0.0,
weight_decay_exclusion_list=[],
gradient_clipping=-1,
do_compile=False,
num_workers_train=1,
save_every=1000,
seed=412,
reload=False,
reload_model_only=False,
strict_reload=True,
track_every=1000,
n_test_batches=20,
device=[0],
use_amp=False,
out_dims=9