Can I Use ChatGPT AI as a GMAT/GRE Tutor?

From Rowan Hand, SWC's in-house GMAT tutor.

Short answer: yes, you can. Just like you can also punch yourself in the face, yet it’s best you don’t.  

There’s no argument that generative AI has been an earthshaking change to the way that we do business and life. 

Most intoxicating, perhaps, is that AI tends to be correct about many things. Its logic and reasoning is not perfect but consistently improving. However, when it comes to facts, there’s an entirely different story to be told. As New York Times technology writer Kevin Roose put it, he thinks of AI as “a very smart assistant who has a dozen Ph.D.s but is also high on ketamine like 30 percent of the time.” 

Add to this the fact that the “hallucination” problem seems only to be getting worse with time. OpenAI, for their part, claims not to know why, but my working speculation is that it has to do with the increased frequency of LLMs (ahem) consuming their own output–much of which is hallucinated. 

Now for the GMAT: I asked ChatGPT to help me study for the GMAT to address when it is and is not advisable to use LLMs to assist with your research or study. I’ll also give you some good rules of thumb for how to navigate the waters of today’s GMAT preparation world effectively.

Sam

Jon

Zack

Alex

Malvika

Rowan

Nanako

Mario

Using AI for Your GMAT/GRE Test Prep

AI systems do tend to be correct about factual information a fair chunk of the time. That makes them extra dangerous, because if you check the factual output by random sampling, you might pick a few correct data points and infer that it’s all correct.  

And yet, hallucinations often pretend to be data-based – and with confidence, at that. Unless you painstakingly fact-check every bit of data you’ll never know how true it is. 

Importantly, the core factual information that AI spits out regarding the GMAT is often wrong. 

There are a number of reasons for this problem, but most specifically it is linked to the GMAT structure and content going through a dramatic change between 2023 - 2024. As far as the AI accesses older data, some GMAT-related information will naturally be out of date.

The new GMAT Focus has a different scoring algorithm from the GMAT Classic. Importantly, the scores are not directly comparable. Any current GMAT score ends with the number 5. All scores from the retired GMAT Classic end with 0. 

This is imperative: be sure not to trust any score information you read that gives scores ending in 0. Be warned: this also goes for information you find outside of AI.

Another major change is that the current scoring system has made the scores look lower.

The median score for the GMAT classic drifted up to approximately 590 in the last full available year of GMAT Classic data (2023). As of now, the median score is lodged somewhere between 555 and 565. Visually, this doesn’t appear to be a dramatic change. 

Where the issue begins is that the former 700 score–the “successful attempt at the GMAT” bar–is now set at approximately 645. A 705 score on the GMAT today reflects approximately a 760 on the GMAT Classic.

Now you have to work harder to get a score that looks lower. 

But there are still a lot of people who imagine that 700 is the bar they’re looking for.

Remember, a lot of online information was published prior to the change and hasn’t been updated. If the data is modern, there’s a good chance that it’s AI-created and factually incorrect. It’s no wonder that the AIs are confused. 

To make matters worse, many schools still suggest GMAT Classic scores as target scores. The M7 and equivalent schools know better, of course, but it takes a lot of time and momentum to change the mental goal that has been in people’s minds for over 30 years. 

To paraphrase Kuhn, I suspect we will need to see a whole generation of admissions officers die off (to be replaced by those hired in 2024 or later) before this problem is rooted out. Of course if the world insists on using LLMs as search engines, the problem will likely never go away. 

More shameful yet, unchecked, uncorrected AI slop proliferates even on many MBA-specific websites. Even stuff that seems largely fact-checked–or at least has been massaged not to read like the robot wrote it–can often be confused as to the current facts. No judgment: there were a lot of changes and the explanations didn’t roll out as effectively as possible. However, if it looks like the AI wrote it, make sure you double-check the numbers. 

When you’re looking at information about the structure of the GMAT exam, I suggest that you only refer to mba.com, which is the official site of the GMAT exam. The information is kept as up to date here as reasonably possible. For score concordance, please refer to the Official GMAT-to-GMAT Classic Score Concordance Chart here (or check for an updated version).

As a final note, even asking an LLM to convert between a GMAT score and a GMAT Focus score is a poor idea. Here is a screenshot of ChatGPT giving information that directly contradicts the GMAT-to-GMAT Classic score concordance charts published by GMAC (the publishers of the GMAT exam). 

Next, let’s explore how generative AIs fare with the different sections of the GMAT.

GMAT Quant and ChatGPT

TL;DR: you probably don’t want to have AI solve Quant questions for you, but it may well be useful to explain certain quant concepts. 

The AI will often (but not always) choose the correct answer.

Admittedly, AI systems are getting much better at solving GMAT questions in the first place. For questions that are fairly easily pasted in, I’ve found the AI to be able to select the correct answer more or less 9 out of 10 times. 

I handed it questions I know to be easy and questions I know to be difficult. The ones with tricky language (rather than tricky math) tended to be the ones that it stumbled on. In short, although the answer itself is likely correct, this needs to be checked with the back of the book. 

It seems from this testing that the longer or more convoluted the language of the question is, the more likely the AI is to misinterpret. Given that 645+ questions often involve complicated concepts as well as complicated wording, this is dangerous ground on which to use AI. 

There are also many questions that will not copy and paste properly because of math formatting: roots, absolute values, fractions, powers, and many other mathematical symbols often copy/paste incorrectly or not at all. If you’re trying to ask AI a question with curious formatting, make sure that it is pasted in correctly.

Furthermore, even when the AI solves the question correctly, its methods rarely–if ever–take advantage of the tricks that the GMAT builds into questions. It tends to brute-force or ludicrously overcomplicate solutions that are designed to be quick and elegant.

In the test center, however, you have a limited time window and rely on a brain with finite energy reserves. The only way to get a high score is to develop techniques and shortcuts to let you solve these questions quickly and effectively. 

Your time will be most effectively spent learning and identifying such shortcuts, NOT asking the AI for help. 

Formulae and Textbook Principles

Some GMAT questions are solvable with a nice high school textbook equation, but many aren’t. As the questions become more sophisticated, they rely on a firm understanding of the mathematical principles that underlie formulae.

The key to GMAT is to understand the core meaning of the formula–can you explain in words what the formula is trying to do?--and do that thing with the numbers you’re given.

For example, look at the following (relatively simple) problem:

Seven paintings are entered in a contest that offers two prizes, and six drawings are entered in a contest that offers one prize. How many possible groups of three prizes exist?

One could blindly plug in to the n!/(r!(n-r)!) formula twice and multiply them together. 

Or you could simply ask yourself how many spaces for the paintings? Two. How many choices? Seven. Does order matter? No, so divide by factorial of group size. Then do the same for the drawings, then multiply the two numbers together.

Remember, the GMAT rewards understanding first principles and asking yourself the right questions. This is often dramatically faster than the plug-and-chug approach; it is also more robust: the plug-and-chug method assumes that numbers will fit easily into formulae, while a first principles approach can adjust to accommodate awkward data, like you’ll get in the GMAT.

Elegant Solutions

GMAT questions are engineered to be vicious-looking but solved with the right “trick.” Add to that the fact that standard math education often uses unnecessarily long, boring processes. And, finally, there are questions that can be answered in mere seconds if you’re simply able to see the forest for the trees. 

For example: 

Each of the 60 books in a library is written in French or German, and each of the books is either a hardcover or a softcover. If a book is to be selected at random from the books in the library, is the probability less than ½ that selected book will be a hardcover written in French?

1. There are 45 hardcover books in the library.

2. Of the books in the library, 24 are written in French. 

The key to the question is this: given that fewer than half of the total books–the Superclass–are in French, it is impossible to have a probability greater than ½ that the book is in French at all. That is, the language split of the total number of books, totally irrespective of whether they be hardcover or softcover–the Subclass. 

Therefore, Statement 2 alone is sufficient based on the Superclass only.

Now I fed ChatGPT the original version of this question from the Official Guide and my rewritten version (as above). In both cases, it was unable to see the Superclass/Subclass distinction. It selected Statements 1 and 2 together as the correct answer.

In such a case, AI – even with its Big Data – is unable to see the big picture. Oops.

To reiterate, the GMAT prefers clean, big picture thinking. The GMAT test taker is better off focusing on first principles, shortcuts, and rules of thumb to answer the questions rather than the (often unnecessary) formulas, convoluted substitutions, or awkward solutions that the AI suggests. 

GMAT Verbal and ChatGPT

TL;DR: RC performance is reasonably solid; CR is still dodgy, but AI loves to give you good arguments for the wrong answers!

It is vital always to check the Official Guide answer. Obvious as it may seem, if the AI’s answer differs from the book, you must trust the book’s answer over the AI’s.

Of course, the Official Guide answer–in principle–tracks with the actual GMAT exam’s reasoning; the AI’s answer does not. 

Reading Comprehension

That said, I’m quite pleased with how the AI has tended to do with Reading Comprehension. This probably has to do with the nature of RC questions themselves. It helps to understand that human readers often misinterpret answer choices because we tend naturally to read language with wiggle room. A lot of GMAT answer choices will use absolutes such as “always” or “never” or superlatives like “best,” and other such language games. The human test taker thinks that maybe that’s not what the question actually means. 

Wrong: the GMAT takes its words very seriously. 

For example, one would always take an answer choice that states “this seems to be a good option” over “this is the best option” in an RC answer. The former is reasonable and hard to disprove, whereas the latter can easily be denied–-as long as we can imagine at least one better option.  

As far as abiding the stricture of the language, this is an excellent place to use AI. As long as the AI explains its reasoning clearly–and that’s one thing it is quite good at, but make sure to ask it to do so!--then having a dialogue with the AI about Reading Comprehension could be a very beneficial tool for you. 

Nevertheless, remember that the AI is using the entire indexed Internet as its data source, so it is invariably reading posts on GMATClub and other forums where GMAT experts like me as well as civilian test takers explain why a particular answer is correct. 

Critical Reasoning

But, the AI only got about 80% correct on Critical Reasoning questions in my testing. Presumably this is because CR questions are more about actual reasoning skill than semantics. 

AI also loves to provide rather compelling explanations as to why its incorrect answers are better than the book’s answer. 

The solution here – you guessed it – is to verify your responses against the back of the book. 

If you’re so inclined, you could even start a dialogue, e.g. “the answer key says that it’s C, not D. Please justify why C is the best answer.” Your mileage may vary. 

GMAT Data Insights and ChatGPT

Data Sufficiency

With standard Data Sufficiency, these questions are often answered correctly. My hit rate, with both formatting-light and formatting-heavy questions was more or less 9/10 correct. 

But all of the same issues apply here: shortcuts are ignored, and answers are often brute-forced. 

Other Types: Table Analysis, TPA, MSR, Graphics Interpretation

The only workaround I was able to find to get the LLM access to the sum total of information in these types of questions was to paste in the URL of a GMATClub page for the question.  

When the answers came back, two things were clear: 

1) The AI missed questions at a higher rate than for either Quant or Verbal; 

2) The AI was reading all of the available tabs and graphics, etc. from GMATClub (which is actually good). 

In my experience, it seemed that the AI’s incorrect answers were often because the AI hallucinated information to provide concrete data. This is definitely problematic, as many questions hinge on the fact that certain information is meant to be inconclusive. 

Bear this in mind: 

First, GMAT Data Insights questions ask you to highlight ambiguous or inconclusive situations and determine what you can and can’t say based on information that is poor or incomplete. That’s the point of the question. 

However, the method by which current generative AI works is to attempt to fill in missing bits of data with an actual value or piece of information. 

Second, the testing sets I used for non-DS Data Insights are really old (circa 2012). That means there is a lot of documentation online about what the correct answers actually are. 

If the AI is getting these wrong, it’s not doing a good job of trawling the internet for precise information. This begs the question of how accurate any specific factual data AI gives you actually might be.

So, again, make sure you check the AI’s answers against the Official answers. 

How to Succeed at GMAT/GRE Study

Now that we’ve talked about when to use (or not use AI), let’s talk about how you can actually study for the GMAT effectively. Here are some tried-and-true methods to help you increase your GMAT score. 

1. The best way, of course, is to work with a reputable GMAT tutor. This will help you avoid obvious pitfalls, track your process, and stay accountable. Many tutors keep a stocked library of quality GMAT materials so you won’t have to worry about much past buying a copy of the Official Guide.

   
   

   Cons: GMAT tutors are, of course, a significant investment. If you find one that isn’t, run.  

2. If cost is a consideration, there are numerous online courses that are available for a reasonable cost (at least compared to a tutor). Some of them are better than others. You can see more at this link. 

   
   

   Cons: Less accountability than a tutor; it’s easy to spin your wheels–or even set yourself back–if you use a poor quality program. The better courses have reasonably good questions; however, you must always use Official questions as well so that you’re comfortable with the writing style.

3. Stick to Official GMAT Questions as much as possible. There are enough Official questions out there for anyone who’s not shooting for 665+ (720-730 on the GMAT Classic). If you are aiming for higher, you can use the Manhattan Advanced GMAT Quant (Quant only) or the Official Advanced Questions book (2019) (Quant and Verbal) to get a good number of extra-tricky questions. 

4. DO NOT ask the AI to create sample questions. Any time I have attempted this, the AI has provided either a softball question, one written in the wrong style, or one that addresses some sort of topic/logic that is outside the GMAT curriculum (note that these categories are not mutually exclusive).

In short, seek Official GMAT questions and spend your time on those. A tutor can help you with this. If you prefer to work with an online course, the big reputable ones have acceptable questions and an interest in keeping things up-to-date. These questions might be written in a slightly different style compared to Official ones, so make sure you’re also practicing with Official questions. 

The point is this: just because AI says it doesn’t make it correct – whether the selected answer or the method. Always verify your answers with the book. The more that you let generative AI lead your test prep, the more you’re going to struggle in the GMAT test center. 

---

Want to work with Rowan on your GMAT prep? Book a free 20 minute introduction chat here.