The city also announced it was considering renaming Lee and Jackson Parks, which both commemorate Confederate war generals.
The crowd that surrounded the statue at Lee Park on Saturday reportedly chanted "You will not replace us" and "Blood and soil," a phrase that popularized in Nazi Germany. But demonstrators denied they were promoting racism, claiming they only wanted to preserve their identity and heritage.
The torchbearers were reportedly led by a right-wing radical Richard Spencer, who joined them after protesting at a similar rally in Jackson Park hours earlier.
"What brings us together is that we are white, we are a people, we will not be replaced," he reportedly said at the rally that took place in the town where he once attended the University of Virginia.
Meanwhile, a group that is suing the city of Charlottesville to keep the controversial statue in place quickly denied its involvement with the protest in a Facebook post.
"We soundly and completely reject racism, white supremacy, and any other identity based groups that preach division and hate no matter which side of the issue they happen to support," the post read.
The rallies drew condemnation from the state officials. Charlottesville Mayor Mike Signer compared the event to the Ku Klux Klan practice of intimidating people of color by setting aflame torches and crosses.
"This event… was either profoundly ignorant or was designed to instill fear in our minority populations," he said in a statement.
The demonstration didn't last long. About 10 minutes in, Charlottesville police arrived at the scene after an altercation between Spencer's group and counterprotesters, breaking up the gathering.
On Thursday, the Louisiana city of New Orleans removed a statue Jefferson Davis, the president of the Confederate States of America during the Civil War.
