Yolanda Foster's ongoing health crisis took a toll on her marriage to David.
The couple, who announced Tuesday they will be divorcing, began growing apart following the Real Housewives of Beverly Hills star's diagnosis with Lyme disease, a source tells PEOPLE.
"Things really changed in their relationship as Yolanda got sick, and then when her health worsened," the source tells PEOPLE. "David travels all the time and they simply weren't spending any time together."
David, 66, and Yolanda, 51, were wed at fairy tale wedding on Nov. 11, 2011. Yolanda was diagnosed with Lyme disease in 2012 and has spoken out about the challenges she's faced, including at one point losing the ability to read and write because of the illness.
"They just grew apart and the intense stress over her health and the disease was very hard on their marriage," the source adds.
Yolanda has three children from her first marriage to Mohamed Hadid: Gigi, 20, Bella, 19, and Anwar, 16.
Foster has been married four times and has five daughters.
"Sadly we have decided to go our separate ways," the couple told PEOPLE in an exclusive statement Tuesday. "We've shared nine beautiful and joyous years together. During that time we experienced love, friendship and the inevitable challenges that come with managing a marriage, careers, blended families and health issues."
Yolanda has chronicled her battle with Lyme disease and fight for recovery on her Bravo blog. "My world has become very small," she wrote in April. "I have moved into an apartment with my husband and son in order to have a more manageable scope of responsibilities while I recover slowly and learn to take life one day at a time. Getting well is my full-time job these days. "