The most comprehensive bank Interview Answer Review tooling available online.

An attention model is an example of a hyper network where the weights of model are determined by the input itself. In the attention mechanism, this occurs in that each token of the input sequence is compared with all others in the context window to determine the next token. The Attention mechanism by default does not care about the order of the input, which is ironic because of the success it has found in next token prediction. This is the basis for the LLMs. The prompt (which may be modified on the backend) will be used and then the next token will be predicted for the answer, and then the answer is built up token by token. 

In the sigmoid function when we pass on larger inputs the function outputs values saturated to 1. Due to this the gradients at those points are also very steady and close to zero. This causes a problem when we are backpropagating through many layers as more and more term between 0 and 1 will multiply and in turn our gradients will continue to decrease. This causes a problem in parameter updating. As gradients are so low, the parameters get changes very slowly and the training process is lengthened.

On the other hand, if we use a relu function, the problem of vanishing gradients is not there as the slope of the function is discontinuous which means for  inputs lesser than 0 the slope is 0 and for inputs greater than 0 it is 1 so due to this at larger input parameters to the activation function does not cause a problem due to consistent slope.

CNN or graph convolutional neural networks is a type of NN that is usually used in vision tasks. Each layer of CNN contains a multilayer filter, where each filter is a k by h matrix, the filter then applied on a M by N matrix as a sliding window, the operation can be a dot product or a pooling layer,