If the compost isn't finished you probably should only use some of it for top dressing. Compost doesn't belong in the hole when you plant a tree.
Your tree will do better if you can plant it in just native soil. If the area you've excavated from under the gravel and pavers is soil, then I would use a mattock or something to break it up a bit (be sure it will drain and not act like a sink to drown your tree). Better than supplementing with potting soil, bring in more native soil to mix with what is underneath when planting the tree. And compost doesn't need to go into the hole. The best thing in the hole is native soil, and
be sure you don't plant the tree too deep, with compost used as top dressing over the dirt.