Using BERT for encoding

[samarzeit]

Dear Authors,

Firstly, I would like to appreciate your excellent piece of work. My question extends to the question from the issue named "Segment Pooling Implementation #2 ", I had the same question and saw this issue. Could you please elaborate where exactly you are doing this encoding part and getting the representation of the entire document? Looking forward to hearing from you.

[chaoweihuang]

Hi,

I appreciate your interest in our work!
We split the documents into chunks and prepare the input_ids like I mentioned in Issue #2
So the shape of the input_ids is (batch_size, num_chunks, chunk_size).
The actual encoding and aggregation happen in modeling_bert.py for the BERT model. You can find the implementation below:

PLM-ICD/src/modeling_bert.py

Lines 86 to 107 in 4106550

	if "cls" in self.model_mode:
	pooled_output = outputs[1].view(batch_size, num_chunks, -1)
	if self.model_mode == "cls-sum":
	pooled_output = pooled_output.sum(dim=1)
	elif self.model_mode == "cls-max":
	pooled_output = pooled_output.max(dim=1).values
	else:
	raise ValueError(f"model_mode {self.model_mode} not recognized")
	pooled_output = self.dropout(pooled_output)
	logits = self.classifier(pooled_output)
	elif "laat" in self.model_mode:
	if self.model_mode == "laat":
	hidden_output = outputs[0].view(batch_size, num_chunks*chunk_size, -1)
	elif self.model_mode == "laat-split":
	hidden_output = outputs[0].view(batch_size*num_chunks, chunk_size, -1)
	weights = torch.tanh(self.first_linear(hidden_output))
	att_weights = self.second_linear(weights)
	att_weights = torch.nn.functional.softmax(att_weights, dim=1).transpose(1, 2)
	weighted_output = att_weights @ hidden_output
	logits = self.third_linear.weight.mul(weighted_output).sum(dim=2).add(self.third_linear.bias)
	if self.model_mode == "laat-split":
	logits = logits.view(batch_size, num_chunks, -1).max(dim=1).values

Best,
Chao-Wei

[muizz-IHT]

Hi, can you provide the code on how to use the model, i have a task to generate cpt and icd codes from an input query, any help would be much appreciated

[samarzeit]

I can't provide my code as I signed an NDA. But I can tell how I did it. I had to submit my work, so to avoid too many documents, I simply copied and pasted the necessary code from different files to a Jupyter notebook. You just have to spend some time going through the repo (I know it is hard) to see what is needed.