Yes. Many forms of OM will continue to decompose changing the level of the soil. Plus you can't work it much w/o it separating the OM and sand. I'm assuming you are leveling for reel mowing. If you are just filling in ruts, holes or gullies that would be a different situation IMO. I use sand for that and I do add OM (peat moss). I find it to take little effort to revisit a few holes that settle and top them off.
There are a few cool season folks that reel mow, but the only one that I'm sure has leveled is
@wardconnor. He'd be a good source for advice on this.
Or maybe one of the warm season guys like
@Ware would give you advice on this.
Or
@Pete1313 
:thumbup: