Well it seems more than likely that it went in the 1860s, having had a good look at maps today. Quite possibly burned down, as it's demise seems a bit quick.
If anyone is local to the site, it might be worthwhile having a look around the back of 46 High Street, as I did find what looked like the remains of an industrial building with the odd charred timber as well. It looked like one retaining wall, possibly one wall of a four storey square base which would have been quite tall, and loads of unidentifiable junk, including bits of iron and shafting. Ken Kirsopp took photos in the 1990s, and this one of them, from the Mills Archive.