"He passed away peacefully after a short battle with cancer," his daughter Emily Gillman told nine.com.au.
"He was 84 so he had a fantastic life."
According to the Daily Mail, Jimmy was in palliative care in recent weeks.
His family, including wife Joanne and daughters, Emma and Melissa, were by his side.
It is believed a private ceremony for the star will be held this Friday.