Hello there, I think I have found the building you were looking for: http://latimesblogs.latimes.com/culturemonster/2008/09/close-reading-l.html
Apparently it had a revamp after you last saw it. I think the original design looks really grim. Today it’s an art museum.
Q: Hello, have you found out what building it was?
A: Not yet. I remember us walking from Battery Park back to our Hotel in 5th Ave. I remember the building I photographed as “modern” 70’s style, concrete colored (I’d say it was pure concrete, no paint on it), lean bauhaus / cubistic style - rounded edges. No Windows, but ventilation grids. The building drew my attention because it didn’t fit in it’s environment - Manhattan is filled with historic buildings, mostly with classic facades. This concrete building just did not fit in it’s surroundings.
I’m sure would recognize it, if I see a picture.
We were walking on the left sidewalk from Battery Park towards Central Park. The building was across the street on the right - the guard had to cross the lanes. There wasn’t so much traffic, which leads me to think we were neither on Broadway, nor 5th Ave but on a parallel street. I remember some details, but it’s hard to make a picture out of it. Humans are unreliable witnesses…