Add pquota to the mount options for your test file system. You will have to remove group quotas if set as they are not compatible with group project quotas.

Create target directories

# mkdir /mnt/xfstest/a /mnt/xfstest/b

Create /etc/projects

33:/mnt/xfstest/a
33:/mnt/xfstest/b

Create etc/projid

testproject:33

Update the projects state and set project limits.

# sudo /usr/sbin/xfs_quota -x /mnt/xfstest
xfs_quota> project -s testproject
xfs_quota> print
Filesystem          Pathname
/mnt/xfstest    /dev/hdb1 (pquota)
/mnt/xfstest/a /dev/hdb1 (project 33, testproject)
/mnt/xfstest/b /dev/hdb1 (project 33, testproject)
xfs_quota> limit -p bsoft=10m bhard=20m testproject
xfs_quota> quota -vp testproject
Disk quotas for Project #33 (testproject)
Filesystem         Blocks      Quota      Limit  Warn/Time      Mounted on
/dev/hdb1            0         10240      20480   00 [--------] /mnt/xfstest