The location of the James' Family Plot was a mystery for many years, and even today there are questions surrounding it. The records of Samuel's burial, state that he was buried on his farm six and a half miles south of West Palm Beach, and Fannie's Death certificate only states that she was buried in Lake Worth. Her obituary in the Lake Worth Herald states that Fannie was buried next to her husband, and that they would be later moved to the Lake Worth cemetery. There was however, no cemetery in Lake Worth at that time. The city purchased land for a cemetery in 1915, however there were no recorded burials there until 1923. There is no official record that the bodies were ever removed from the plot.

It is known that the burial plot was 10ft by 10ft, and that it was originally surrounded by a wrought iron fence. In the 1920's, a group of men came in the dead of night, with shovels and lanterns to the plot, and tore down the fence and removed the markers, leaving no trace. The event was witnessed by Jean Childs Addison, when she was a child. When she told the story later in life, she did not mention whether the bodies had been dug up or not. Eventually no one remembered that the plot had been there at all.

Recent research has discovered records that provide evidence of the actual location. When Fannie sold the land around the plot, the plot itself was exempted from the sale. The legal description of the land indicates that the plot was located in the 300 block of L Street. Ground penetrating radar was used in 2013 to scan this area and that confirmed that the ground had been disturbed in the past, in a way that is consistent with burial sites, but could not confirm whether there were bodies here or not.