What Agents Structurally Struggle With