Seems like the railroad service trucks (the ones with the lowerable steel wheels to fit the rails) do it all the time. Rails spacing is less than typical vehicle tracking width.

Might be an issue long term with the change in loading on the wheel bearings. They're probably designed for the load to be where the factory wheels would put it, and by removing one wheel, you're shifting the load in or out significantly. That is always a discussion topic with the 4-wheeler groups I frequent, cause to fit the larger tires on, wheel spacers are often used.