Yeah, families can be a kick in the butt. I have one older brother and one older sister. Growing up my sister was always my protector, and my brother was not much of an older brother, at least not in a good way. We shared a room, he is almost 6 years older, and he never wanted siblings, and pretty much lived like he was an only child. Completely selfish and self absorbed, but a nice friendly guy, whom everyone seems to like.
After 30+ years, my coming out & my brother getting divorced from a wife whom we all loved dearly, we seemed to develop an appreciation for each other, and the right for each to be happy. We still have almost nothing in common, but have many memories of family vacations & other experiences of growing up, and enjoy each others' company.
My sister has become a right-wing, born again christian, angry nut job, who is openly hostile and always looking for an excuse to "go off" on someone. Unfortunately, she has passed this on to her 2 daughters, who are now adults and also miserable terrors. Sadly, I have had to chose to have no relationship with them at all. After years of trying to change her, to no avail.
My brother was diagnosed with ALS (aka Motor Neuron disease and Lou Gehrig's disease) last May, and now I need to make more trips back home (which is 2500 miles from California). So far I have been able to go when my sister is out of town, and avoid all the drama she & her children cause, but I realize that as my brother's condition deteriorates, I will not get to be so selective, and will have to run into them.
It makes a very tough situation, even more terrible.
Try to always keep the door open, as things can change. Unless, of course, they are purposely trying to hurt you. Which I doubt is the case, or you wouldn't live across the street from them.