The Commonwealth Games Federation (CGF) struggled to find a host before Victoria volunteered in April 2022.
But the premier said the projected cost had now tripled and become “well and truly too much” for the state to bear.
CGF called the decision “hugely disappointing” and said it is “committed to finding a solution”.
The Commonwealth Games are a multi-sport tournament that take place every four years. They have only ever been cancelled during World War Two.
To be eligible to participate in the games, competitors must be from one of the Commonwealth’s 56 members. Most of the countries in the Commonwealth were once part of the British Empire.
Victorian Premier Daniel Andrews on Tuesday said Victoria had been “happy to help out” when approached to host last year, but “not at any price”.
Organisers had originally estimated the event – hosted across cities including Geelong, Bendigo and Ballarat – would cost A$2.6 billion (£1.4bn; $1.8bn) and the state government had billed it as a boost for the regions.
But now the 12-day tournament was expected to cost more than A$6 billion, Mr Andrews said, adding that the new figure was “more than twice the estimated economic benefit” it would bring to Victoria.
“I’ve made a lot of difficult calls, a lot of very difficult decisions in this job. This is not one of them,” he told a press conference.
“That is all cost and no benefit.”
The government will still complete the stadium upgrades it had promised ahead of the games, while using the money it is now saving on housing and tourism initiatives.
Mr Andrew said the government had considered “every option” including moving the games to Melbourne, before informing CGF of their decision.
“Amicable and productive” meetings had occurred in London overnight, he added.
But in a statement on Tuesday, the governing body said they were blindsided by the decision.
“We are disappointed that we were only given eight hours’ notice and that no consideration was given to discussing the situation to jointly find solutions,” it said in a statement.
CGF said the estimate of A$6 billion is double the figure they were advised of at a board meeting last month, and that the increase in costs were due to the “unique regional delivery model” that Victoria chose for the games.
The government had made decisions to include more sports and changed plans for venues, often against the advice of the CGF and its Australian arm, all of which added “considerable expense”, it said.
Victorian Opposition Leader John Pesutto said the decision is a “massive humiliation” for the state, and “hugely damaging” for its reputation as a global events leader.
The chief executive of Commonwealth Games Australia agreed, saying the cited cost blowout was a “gross exaggeration” and that it would try to find another Australian host for the 2026 games.
Australia has held the Commonwealth Games five times – including on the Gold Coast in 2018 and in the Victorian capital of Melbourne in 2006 – but all of the country’s states on Tuesday ruled out picking up the event.
New Zealand authorities also say they will not take on the games.
Organisers have had great difficulty finding viable tournament hosts in recent years.
The South African city of Durban was originally set to stage the 2022 games, but were stripped of hosting rights in 2017 after running into money troubles and missing key deadlines. Birmingham agreed to host nine months later.
CGF had originally hoped to name a host for the 2026 games in 2019, but several hopeful bidders withdrew from the process due to cost concerns, leaving it unable to lock in Victoria as the hosts until 2022.