Prince Harry and Prince William could have been a “brilliant” team.

The brothers — along with Harry’s wife Meghan Markle and William’s wife Kate Middleton — were thought to be the “Fab Four” until a rift between them led to a separation of their royal offices and Harry and Meghan’s eventual exit from the royal family.

“It’s a sad state of affairs because William and Harry could have been so brilliant” as a team, says a close insider in this week’s issue of PEOPLE. “To think of what they could have achieved together is almost heartbreaking.”

While William, 38, and Harry, 36, were always on “different paths” due to Prince William’s future role as king, they grew up close — especially after the death of their mother Princess Diana when they were young.

Max Mumby/Indigo/Getty Images Prince Harry and Prince William

Paul Grover – WPA Pool/Getty Images Queen Elizabeth, Meghan Markle, Prince Harry, Prince William and Kate Middleton

On July 1, the brothers are set to come together for the unveiling of a statue in honor of their mother at Kensington Palace. While both William and Harry (who relocated to California last year) are determined to keep that appointment, they will have to talk things over first.

“Neither of them have good poker faces,” a palace source says.

Getty Prince William, Princess Diana and Prince Harry

During Harry and Meghan’s interview with Oprah Winfrey on March 7, the Duke of Sussex said that other members of his family can’t leave the “system” the way he did.

“It’s been really hard,” Harry explained. “I am part of the system with them, I always have been, but I guess — and I’m very aware of this — my brother can’t leave that system, but I have.”

Tim Graham Photo Library via Getty Prince William and Prince Harry in 2002

In a new report from The Sunday Times, a source close to both brothers claims that William, who is second in line to the throne behind their father, found those comments “way off the mark.”

“He has a path set for him and he’s completely accepting of his role. He is very much his grandmother’s grandson in that respect of duty and service,” the source told the outlet.

Joe Pugliese/Harpo Productions Prince Harry, Meghan Markle and Oprah Winfrey

Prince Harry also described his relationship with his older sibling at the moment as “space.”

“I love William to bits. He’s my brother. We’ve been through hell together,” Harry explained. “But we are on different paths.”