One video multiple class

[akash16iitkgp]

Problem Statement:
I have to build a solution to track human activity like sleeping/waking up/yawning etc through a single video(video regognition.
For eg a camera is recording a person and will pass the feed continously to the model. The model will classify the action.

Approch:
I cropped small clips from big video to train a slowfast classification model but unfortunately while predicting whole video is classified as one class and although I trained for only two classes it's showing top 5 results.
top 5 [(1, 0.5401362180709839), (0, 0.45834362506866455), (28, 3.901471245626453e-06), (138, 3.874579306284431e-06), (12, 3.874179128615651e-06)].

My question is why it is predicting top 5 classes?
Another question is how the prediction class will change as diffenet activity is performed in video?
time-stamp ground-truth predictions
0--30 sec sleeping sleeping
30-40 sec waking up sleeping

The confidence is distributed when I have video performing two activities as above