So as I said, when you upload an object, the collision boxes around it are getting created automatically, so you can select and collide with the object. But there is a little workaround to achieve what you want to do.
You can find the ID of the castle from the inspector and enter it in the brackets.
Please keep in mind that this is an internal method and can change at some point.
Now you can enter your castle but not collide with the walls anymore. So you can use some invisible cubes and place them at the position of the castle walls.
I hope that is helpful