Should I use a sequence input layer or an image input layer for a combined CNN/LSTM neural network?

I am attempting to use a CNN/LSTM to take in a series of frames from a video of two liquids mixing together to predict their viscosities.

My initial layout is shown in the attached image and I planned on seperating a cell array of frames into stacks of sequences to use as inputs.

I was told that this would not work and an alternative approach is to use 2D or 3D (not sure which) image input layers and then use time as a seperate input for the LSTM portion. I'm not sure I understand what this means or why my approach was said to be wrong.

Which, if any, approach is best? Also, if neither of them are, is there a better method?

[Matt J]

Another possible approach is to use 2D or 3D (not sure which) image input layers and then use time as a seperate input for the LSTM portion.

I don't think I understand this alternative approach. What does it mean to "use time as a separate input"? You want the network to regard each input as a sequence of 2D images, correct?

[Jade]

Matt,

To be honest, I'm not sure I understand the alternative approach either, but it was suggested by one of my professors and he admitted he could be wrong. Figured I'd mention it in case someone more familiar than me came along and had a suggestion.

Yes, I want the network to regard each input as a sequence of 2D images, that's correct. Here's a screenshot of the analyze network tool as well.

[Matt J]

OK, well it doesn't look like network analyzer is showing any errors. Is there something that's not working?

[Jade]

Matt,

It seems to run with 1 video so far, and I'm in the process of scaling it up now. Training loss returned NaN at first, but adjusting the learning rate seems to have solved that issue.

Just wanted to make sure this was the correct approach to the problem before going too far in the wrong direction. There's a lot of different network structures and I'm still learning. Really appreciate the help!