The rectangular waveguide, being a waveguide and a one-conductor propagating transmission line, can propagate only at higher-order modes. The modes of transmission can consequently be only in TE and TM modes. Subscripts are used to describe the electric and magnetic field configurations. The general symbols of TE or TM are used to describe transverse electric or transverse magnetic waves. The subscript 'm' indicates the number of HALF-WAVE variations of the electric field along the WIDE dimension of the waveguide, and 'n' indicates the number of HALF-WAVE variations of electric or magnetic field in the NARROW dimension of the guide (n for "narrow"). The TE mode, which has the longest operating wavelength, is designated as the DOMINANT mode. This is the mode for the lowest frequency that will propagate in the waveguide. If 'a' (the wide dimension of the waveguide) is less than half a wavelength (< l /2), NO PROPAGATION WILL OCCUR. Therefore, the waveguide acts as a high pass filter. For a rectangular waveguide, the cutoff frequency can be found from:

where : = c = 3 x 108 m/s (speed of light), m and n = the subscripts of the particular TE or TM mode, a and b = the wide and narrow dimensions of the rectangular guide.
The cutoff wavelength is then given by:

The cutoff for the dominant mode can easily be calculated. The dominant mode being the TE10. Substituting 1 for 'm' and 0 for 'n',

That is,
![]()
The field configuration in a rectangular waveguide propagating the commonly used TE10 mode is shown in the applet below.
Rectangular waveguides were originally chosen so that the dominant mode would exist over a certain frequency range. This frequency range determines the 'a' dimension. The 'b' dimension is chosen on the basis of the following criteria:
- the attenuation loss is greater as the 'b' dimension is made smaller
- the 'b' dimension determines the voltage breakdown characteristics and therefore determines the maximum power handling capacity.
For larger power handling capacity, 'a' and 'b' should be made as large as possible. In practice, 'b' is made about half of the 'a' dimension. It is desirable in all types of transmission systems to select sizes so that only one mode of propagation is possible. In other words, the physical size of the guide is related to the frequency band under consideration. Because of the possibility of higher-order modes, it has become common to operate waveguides over an approximately 50% frequency band. By properly selecting this frequency range, it is possible to operate far enough from cutoff so that the guide parameters do not vary too rapidly and to avoid also the frequency region where other modes are possible. The figure below shows the field configurations of some of the other higher-order modes.