I'd recommend about 2-4gb. But in Puppy Linux I got by with 512mb storage/persistence files, and in Puppy you don't have to worry about exceeding it, as it has a program to grow it.
And you can also get a lot of programs in .sfs (squashfs) files, which are mounted to the root file-system. In other words extensions.

Anyways, if you want to use Xubuntu on a usb drive at all times instead of creating an ext4 partition, you will need 3-4gb. Probably. But even if you don't find some program to extend it with, you can create a new ext4 file-system in a file, mount it, set permissions so your user can write to it, and just copy the files from the old persistence file over.

So perhaps just start with a 2gb file, and partition your drive from inside Xubuntu with GParted (just keep your Windows C drive intact), and run the install choosing the partition you made. This way things will load faster as well, especially if you've got an SSD. But Linux requires an EXT, XFS or other Unix file-system for its root partition. Which it can of course get on a VFAT fs as well, if it mounts the file-system file.