cnlpt.CnlpModelForClassification module

Module containing the CNLP transformer model.

class cnlpt.CnlpModelForClassification.ClassificationHead

Bases: Module

Generic classification head that can be used for any task.

__init__(config, num_labels, hidden_size=-1)

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(features, *kwargs)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class cnlpt.CnlpModelForClassification.RepresentationProjectionLayer

Bases: Module

The class that maps from some output from a text encoder into a feature representation that can be classified. Project the representation to a new space depending on the task type, based on arguments passed in to the constructor. :param config - The config file for the encoder :param layer - Which layer to pull the encoder representation from :param tokens - Whether to classify an entity based on the token reprsentation rather than the CLS representation :param tagger - Whether the current task is a token tagging task :param relations - Whether the current task is relation exttraction :param num_attention_heads - For relations, how many “features” to use :param head_size - For relations, how big each head should be

__init__(config, layer=10, tokens=False, tagger=False, relations=False, num_attention_heads=-1, head_size=64)

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(features, event_tokens, **kwargs)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class cnlpt.CnlpModelForClassification.CnlpConfig

Bases: PretrainedConfig

The config class for CnlpModelForClassification.

Parameters:

  • encoder_name – the encoder name to use with transformers.AutoConfig.from_pretrained()

  • finetuning_task (Optional[str]) – the tasks for which this model is fine-tuned

  • num_labels_list (List[int]) – the number of labels for each task

  • layer (int) – the index of the encoder layer to extract features from

  • tokens (bool) – if true, sentence-level classification is done based on averaged token embeddings for token(s) surrounded by <e> </e> special tokens

  • num_rel_attention_heads (int) – the number of features/attention heads to use in the NxN relation classifier

  • rel_attention_head_dims (int) – the number of parameters in each attention head in the NxN relation classifier

  • tagger (List[bool]) – for each task, whether the task is a sequence tagging task

  • relations (List[bool]) – for each task, whether the task is a relation extraction task

  • use_prior_tasks (bool) – whether to use the outputs from the previous tasks as additional inputs for subsequent tasks

  • **kwargs – arguments for transformers.PretrainedConfig

__init__(encoder_name='roberta-base', finetuning_task=None, num_labels_list=[], layer=-1, tokens=False, num_rel_attention_heads=12, rel_attention_head_dims=64, tagger=[False], relations=[False], use_prior_tasks=False, hier_head_config=None, **kwargs)
class cnlpt.CnlpModelForClassification.CnlpModelForClassification

Bases: PreTrainedModel

The CNLP transformer model.

Parameters:

  • class_weights (Optional[List[float]]) – if provided, the weights to use for each task when computing the loss

  • final_task_weight (float) – the weight to use for the final task when computing the loss; default 1.0.

  • argument_regularization (float) – if provided, the argument regularization to use when computing the loss

  • freeze (bool) – whether to freeze the weights of the encoder

  • bias_fit (bool) – whether to fine-tune only the bias of the encoder

config_class

alias of CnlpConfig

__init__(config, *, class_weights=None, final_task_weight=1.0, argument_regularization=-1, freeze=-1.0, bias_fit=False)

Initializes internal Module state, shared by both nn.Module and ScriptModule.

compute_loss(task_logits, labels, task_ind, task_num_labels, batch_size, seq_len, state)

Computes the loss for a single batch and a single task.

Parameters:

  • task_logits –

  • labels –

  • task_ind –

  • task_num_labels –

  • batch_size –

  • seq_len –

  • state –

apply_arg_reg(logits, attention_mask, state)

Applies argument regularization to the logits for a single batch on all tasks.

Parameters:

  • logits (List[torch.FloatTensor]) – the computed logits of the batch.

  • attention_mask (torch.LongTensor) – the attention mask for the batch.

  • state – the state dict containing the loss for the batch.

forward(input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, labels=None, output_attentions=None, output_hidden_states=None, event_tokens=None)

Forward method.

Parameters:

  • input_ids (torch.LongTensor of shape (batch_size, sequence_len), optional) – A batch of chunked documents as tokenizer indices.

  • attention_mask (torch.LongTensor of shape (batch_size, sequence_len), optional) – Attention masks for the batch.

  • token_type_ids (torch.LongTensor of shape (batch_size, sequence_len), optional) – Token type IDs for the batch.

  • position_ids – (torch.LongTensor of shape (batch_size, sequence_len), optional): Position IDs for the batch.

  • head_mask (torch.LongTensor of shape (num_heads,) or (num_layers, num_heads), optional) – Token encoder head mask.

  • inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_len, hidden_size), optional) – A batch of chunked documents as token embeddings.

  • labels (torch.LongTensor of shape (batch_size,), optional) – Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy).

  • output_attentions (bool, optional) – Whether or not to return the attentions tensors of all attention layers.

  • output_hidden_states – not used.

  • event_tokens – a mask defining which tokens in the input are to be averaged for input to classifier head; only used when self.tokens==True.

Returns: (transformers.SequenceClassifierOutput) the output of the model