So the non-member can see the content of the long text attribute, but not the item and the folder name (they show as “Deleted Item” and “Deleted Folder” instead).
In contrast to that, this is what a member of the folder sees:
I went on and tested this on my side and you are actually right, I ended up with the same result, so naturally I contacted our team and confirmed that it is something that shouldn’t be there