Yes. Many forms of OM will continue to decompose changing the level of the soil. Plus you can't work it much w/o it separating the OM and sand. I'm assuming you are leveling for reel mowing. If you are just filling in ruts, holes or gullies that would be a different situation IMO. I use sand for that and I do add OM (peat moss). I find it to take little effort to revisit a few holes that settle and top them off.
There are a few cool season folks that reel mow, but the only one that I'm sure has leveled is @wardconnor
. He'd be a good source for advice on this.
Or maybe one of the warm season guys like @Ware
would give you advice on this.