How to compare two neural networks?
There are several metrics:
- Number of parameters
- GPU occupancy (
- Training/inference time
Here we only talk about
parameters. Note: Floating point of operations (FLOPs) is different with floating point of per second (FLOPS). The FLOPS is the same for the same hardware device, but the FLOPs are different for different networks.
FLOPs is related to network design: Number of layers, activation layer selection, parameters, etc.
The difference between FLOPs and parameters is shown at the top figure.
Because Convolution layer can share the kernel, its parameters is far lower than the FLOPs.
PS: Throughput refers to the number of examples (or tokens) that are processed within a specific period of time, e.g., “examples (or tokens) per second”.