When people talk about rapid stylistic change in rock ‘n’ roll, a period often cited is the five years between 1962 and 1967. Namely, that time during which The Beatles travelled from Love Me Do to A Day In The Life – which saw them change from loveable light entertainers to musical revolutionaries, with outlandish clothes, moustaches and expanded minds.

Nor is five years a bad period of time to measure the career of David Bowie, who died in January. In 2013, a documentary (The title of which, Five Years, is also the name of a 1972 song in which Bowie imagines an imminent apocalypse) explored several pivotal years in his career. In 2015, meanwhile, a boxed set with the same name was released charting Bowie’s activity between 1969 and 1973. It was a time when assuming of new personae actively assisted Bowie’s musical process. He transformed himself from ingenuous folky singer-songwriter to a heavy rocker, a hippy family man and ultimately to Ziggy Stardust, a lurid, satirical and highly effective rock ‘n’ roll star.

Having got to the top of the charts with Ziggy, he then just as abruptly killed off the character. If we can fleetingly imagine box sets as having the same kind of dynamic as a TV series, then this was the end-of-season cliffhanger.

If he could move fast in five years, it’s a testament to Bowie’s transformative powers that this current, similarly-proportioned box set (12 discs of remastered music, including an entire unreleased album and a compendium of rarities) is the product of only two years work.

Clearly, however, that workload didn’t come without cost. In mid-1974, the BBC sent a film crew to follow Bowie’s US tour – clearly hoping to get the whole Ziggy story, but finding an arguably far more interesting one.

Live, Bowie commanded band and audience with unquestionable power. Offstage, the BBC found a waifish, alien creature riding through the desert in a black limousine. Pale and unimaginably fragile, he listened to Aretha Franklin on the stereo, and free-associated answers to interview questions with the aid of a fly floating in a carton of milk.

In another scene, he travels through Los Angeles in a limousine and expounds on the disconnect between the city’s apparent calm and its seething internal tension. As he talks, police sirens blare outside the car. “I hope we’re not stopped,” he says, evidently alarmed. It’s not cold out, but he seems to be sniffing a good deal.

The discs here put the musical flesh on the bones of that reportage, a period in which Bowie junked one project for another, and developed an alarming drug problem (“I was totally crazed,” he later said).

All that notwithstanding, it was also the period when he made a worldwide hit album Young Americans, recorded Fame, a single which connected him for the first time to a black American audience and took a leading role in a major motion picture The Man Who Fell To Earth. He then made yet another new album.

As the box elucidates in its printed materials and the live album David Live, his latest transformation was to become a performer of soul music.

In spring 1974, Bowie arrived in America ostensibly to promote his LP Diamond Dogs – an album of dystopian rock ‘n’ roll made by a man with no eyebrows, dressed in a fishnet stocking. Audiences expecting this character were instead surprised to find themselves facing a Bowie with a wedge haircut, and a band interpreting his music with the aid of saxophones and conga drums.

At a break in the tour, Bowie took further action. He ditched the elaborate theatrical, “Hunger City” set that had accompanied him so far. He then booked time at Sigma, an east coast recording studio famed as the home of the “Philadelphia Sound”, a melodic and tastefully-orchestrated version of disco music, presided over by producers Leon Gamble and Ken Huff. There, he began recording new music.

Surrounded by American musicians, including a young Luther Vandross, Bowie recorded his impressions of the country in its lingua franca: soul. Here, boys seduced their girls into a life inside a flawed, Nixonian version of the American dream.

More revealingly, after three years in character, the artist now seemed to be playing himself, recording songs as confessionally-titled as Who Can I Be Now? It was as if the music of black America, long an enthusiasm, allowed him to access what sounded like a more vulnerable, emotional side.

It would have been a magnificent album: funky, moving and deep. Then, though, Bowie met John Lennon, recorded a couple of songs with him, and dropped the entire thing (mastered, arranged and titled The Gouster it appears here for the first time), confining it to legend. A new collection of tracks, called Young Americans and featuring Fame, the song he cut with Lennon, was left to take its place in the hearts of a public until then unaware of Bowie’s charms. Until they got their own copies, even the musicians who played on the album were unaware the degree to which its tracklisting had changed.

This box helps make coherent a period of Bowie’s career that was low on specifics by virtue of his workload, his lifestyle and his dislike of revisiting the past. This was a time in which days and nights, reality and fiction merged. Art now imitated life: having been impressed with his strangeness in the BBC film, in 1975 director Nicholas Roeg now cast him as an alien, Thomas Newton, in his movie The Man Who Fell To Earth.

If Bowie was Newton, Newton also was Bowie, with an image from the film forming the cover for his next album. Recorded over 12 weeks in Los Angeles after shooting wrapped on the film, Station To Station was written in the studio, but its quality and heft are quite at odds with the mania at the time of its creation. Paradoxical? Let us count the ways.

A coherent work deriving from being psychologically all over the place. The product of outside influence, yet quite its own animal. Essentially, it illustrated how Bowie could envision work as a director might, then rise to the challenge of his own proposition. Presiding over the record was a charming despot, The Thin White Duke. Surrendering to his demands, Bowie then took the Duke on tour, an expressionistic feast for the eyes, in deep black and blinding white. Before the band came on stage, music by the German electronic band Kraftwerk was played. A live album from the tour is the last major document in this collection.

For someone who didn’t like to look back – indeed was moving forward at unintelligible speed – a Bowie archive set clearly presents a problem. This is not, clearly a set like The Beatles’s Anthology or The Beach Boys’s Pet Sounds sessions which simply adds peripheral material to the extant work. Instead of superfluity and works in progress, it does Bowie the service of presenting his transitional work not as sketches but as something else: integral parts of a much larger picture, which is only now gradually being revealed.

John Robinson is associate editor of Uncut and the Guardian Guide’s rock critic.