You have in the feed, 100 kmol/h methane and 21 kmol/h Oxygen.
The reactions are serial, the first reaction
Methane + 0.5 * Oxygen => Carbon Monoxide + 2 Hydrogen
is set to convert 0.4 * 100 kmol/h Methane, so 40 kmol/h methane. In doing so, it also consumes 20 kmol/h Oxygen. Remains, 60 kmol/h methane and 1 kmol/h Oxygen.
The next reaction
Methane + Oxygen => Carbon Dioxide + 2 Hydrogen
is asked to convert 60 percent of the remaining methane (serial box is checked), so 0.6 * 60 kmol/h = 36 kmol/h. In doing so, this reaction requires 36 kmol/h Oxygen, but you have only 1 kmol/h Oxygen left. Hence, the error is
error: calculate failed for unit FixedConversionReactor_5: Conversion of 0.6 for compound Methane for reaction 2 caused negative concentrations of compound Oxygen